Method and apparatus for decimation mode determination utilizing block motion

ABSTRACT

A motion image reproduction apparatus that reproduces motion image data from converted motion image data. The motion image reproduction apparatus includes a block expander that receives converted block data included in the converted motion image data and data indicating a conversion mode in which the converted block data was converted and that expands the converted block data based on the data indicating the conversion mode abd a combiner that produces frame data by combining blocks reproduced via the block expansion performed by the block expander. The block expander reproduces the block by performing the block expansion process in a mode corresponding to at least one of a spatial decimation process in a fixed sampling point position mode, a spatial decimation process in a sampling point position shifting mode, and a temporal decimation process, performed in the production of the converted motion image data. The combiner produces the frame data by combining blocks reproduced via the block expansion process.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a division of U.S. application Ser. No. 11/119,122,entitled “METHOD AND APPARATUS FOR DECIMATION MODE DETERMINATIONUTILIZING BLOCK MOTION,” filed on Apr. 30, 2005, the entirety of whichis incorporated herein by reference to the extent permitted by law. Thepresent invention claims priority to Japanese Patent Application No. JP2004-147739, filed May 18, 2004, the entirety of which is alsoincorporated herein by reference to the extent permitted by law.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to a method and apparatus for convergingmotion image data, a method and apparatus for reproducing motion imagedata, a computer program for converting motion image data, and acomputer program for reproducing motion image data, and moreparticularly, to a method and apparatus for converging motion imagedata, a method and apparatus for reproducing motion image data, acomputer program for converting motion image data, and a computerprogram for reproducing motion image data, capable of performinghigh-quality data conversion such that motion image data is compressedwith substantially no degradation in image quality, and capable ofperforming inverse data conversion such that high-quality motion imagedata is reproduced from compressed motion image data.

2. Description of the Related Art

When motion image data is stored on a storage medium such as a hard diskor a DVD or when motion image data is transmitted via a network, themotion image is generally compressed to reduce its data size. In recentyears, a great improvement in quality of motion image data has beenachieved. For example, an HD (High Definition) format is now availablefor representing high-quality motion image data. However, with improvingin data quality, a drastic increase in the data size is occurring. Tosolve such a problem, a wide variety of techniques to improve efficiencyin compression/decompression of motion image data and to preventdegradation in data quality due to compression/decompression are underdevelopment and evaluation.

A known technique to compress motion image data is spatial decimation todecimate pixels constituting an image frame of motion image data.Another known technique is temporal decimation to decimate frames (toreduce the frame rate).

By performing such data conversion, the data size can be reduced andthus it becomes possible to efficiently store data on storage medium ortransmit data via a network. However, compression of data causesdegradation in image quality. That is, data reproduced from compressedimage data is not as good as the original data. The problem withdegradation in image quality is serious, in particular, when originaldata is of a high-resolution image.

A wide variety of techniques have been proposed to reduce suchdegradation in image quality. For example, Japanese Unexamined PatentApplication Publication No. 2003-169284 discloses an image compressiontechnique in which parameters are set based on information indicatingthe brightness of an image, and a compression mode is switched dependingon the brightness of the image. Japanese Unexamined Patent ApplicationPublication No. 2002-27466 discloses an image compression technique inwhich a screen is divided into a plurality of regions, and an optimumcompression mode is determined for each region.

SUMMARY OF THE INVENTION

However, the known techniques to improve the data quality by selectingan optimum compression mode based on various characteristics detectedfrom image data being processed cannot sufficiently suppress degradationin image quality due to compression/decompression of image data.

Motion image data includes various parts, some of which includes asubject moving at a high speed, some of which includes a subject movingat a low speed, and some of which includes a subject at rest. It isdifficult to properly compress motion image data including such variousparts without causing significant degradation in image quality so thatdegradation in quality of image data reproduced from compressed motionimage data is not substantially perceptible.

In view of the above, the present invention provides a method andapparatus for converting motion image data, a method and apparatus forreproducing motion image data, a computer program for converting motionimage data, and a computer program for reproducing motion image data, bywhich an optimum compression mode is determined depending on a featureof a region of an image, in particular, depending on motion of asubject, and data conversion is performed for each region in thedetermined optimum mode thereby making it possible to compress imagedata and decompress the compressed image data with substantially nodegradation in image quality.

According to an embodiment, the invention provides a motion image dataconversion apparatus that performs data conversion of motion image data,including a block divider that divides each frame of the motion imagedata into blocks, a motion detector that detects the amount of motion ofa subject for each block generated by the block divider, and a blockprocessing unit that receives block data of blocks generated by theblock divider and the data of motion detected by the motion detector andthat decimates the block data, the block processing unit including adecimation mode determination unit that determines a decimation mode asto whether to perform spatial decimation in a fixed sampling pointposition mode or a sampling point position shifting mode, in accordancewith the data of motion, and a decimation execution unit that executesspatial decimation in the fixed SPP mode or the SPP shifting mode inaccordance with a determination made by the decimation modedetermination unit.

In the motion image data conversion apparatus, the decimation modedetermination unit may further determine a direction in which the samplepoint position is shifted with frame advance in the spatial decimationprocess in the SPP shifting mode, and the decimation execution unit mayshift the sample point position in the direction determined by thedecimation mode determination unit when the spatial decimation processis performed in the SPP shifting mode.

In the motion image data conversion apparatus, the decimation modedetermination unit may determine the decimation mode in accordance withdata indicating the correspondence between an evaluated image qualityscore and a moving speed of a subject, the evaluated image quality scorebeing estimated for data obtained as a result of the decimation processin the fixed sampling point position mode, the determination being madesuch that when a moving speed of a subject indicated by the data ofmotion detected by the motion detector corresponds to an evaluated imagequality score equal to or higher than a predetermined threshold value,the decimation mode determination unit determines that the decimationprocess should be performed in the fixed process point position mode,but when a moving speed of a subject indicated by the data of motiondetected by the motion detector corresponds to an evaluated imagequality score lower than the predetermined threshold value, thedecimation mode determination unit determines that the decimationprocess should be performed in the process point position shifting mode.

In the motion image data conversion apparatus, the decimation modedetermination unit may further include a memory adapted to store adecimation mode determined for a previous frame by the decimation modedetermination unit, and the decimation mode determination unit maydetermine the decimation mode in accordance with data indicating thecorrespondence between an evaluated image quality score and a movingspeed of a subject, the evaluated image quality score being estimatedfor data obtained as a result of the decimation process in the fixedsampling point position mode, and also in accordance with the decimationmode determined for the previous frame, the determination being madesuch that when the decimation mode determined for the previous frame wasthe fixed sample point position mode and when a moving speed of asubject indicated by the data of motion detected by the motion detectorcorresponds to an evaluated image quality score equal to or higher thana predetermined threshold value T1, the decimation mode determinationunit determines that the decimation process should be performed in thefixed process point position mode, when the decimation mode determinedfor the previous frame was the fixed sample point position mode and whena moving speed of a subject indicated by the data of motion detected bythe motion detector corresponds to an evaluated image quality scorelower than the predetermined threshold value T1, the decimation modedetermination unit determines that the decimation process should beperformed in the process point position shifting mode, when thedecimation mode determined for the previous frame was the sample pointposition shifting mode and when a moving speed of a subject indicated bythe data of motion detected by the motion detector corresponds to anevaluated image quality score lower than a predetermined threshold valueT2, the decimation mode determination unit determines that thedecimation process should be performed in the process point positionshifting mode, and when the decimation mode determined for the previousframe was the sample point position shifting mode and when a movingspeed of a subject indicated by the data of motion detected by themotion detector corresponds to an evaluated image quality score equal toor higher than the predetermined threshold value T2, the decimation modedetermination unit determines that the decimation process should beperformed in the fixed process point position mode.

In the motion image data conversion apparatus, the block processing unitmay perform the spatial decimation process in the sample point positionshifting mode such that a virtual moving speed of a subject resultingfrom execution of the decimation process in the sample point positionshifting mode falls within a range of the moving speed in which anevaluated image quality score corresponding to the virtual moving speedis equal to or higher than a predetermined threshold value, theevaluated image quality score being estimated based on prepared dataindicating the correspondence between the moving speed of a subject andthe evaluated image quality score of data obtained as a result of thedecimation process in the fixed sampling point position mode.

In the motion image data conversion apparatus, the block processing unitmay further include a memory adapted to store data indicating a mode inwhich the sample point position is shifted or fixed in the spatialdecimation process performed by the block processing unit for a previousframe, and when the virtual moving speed of a subject obtained as aresult of spatial decimation performed on the previous frame dependingon the mode in which the sampling point position was shifted or fixed isevaluated based on data indicating the correspondence between the movingspeed of a subject and the image quality score for data obtained as aresult of decimation in the fixed SPP mode, if an image quality scorecorresponding to the virtual moving speed of the subject is equal to orhigher than a predetermined threshold value T3, then the spatialdecimation may be performed in the same mode as the mode used in thespatial decimation performed on the previous frame, but when the virtualmoving speed of the subject obtained as the result of spatial decimationperformed on the previous frame depending on the mode in which thesampling point position was shifted or fixed is evaluated based on dataindicating the correspondence between the moving speed of a subject andthe image quality score for data obtained as a result of decimation inthe fixed SPP mode, if the image quality score corresponding to thevirtual moving speed of the subject is lower than the predeterminedthreshold value T3, then the spatial decimation may be performed in amode selected such that the virtual speed of the subject obtained as aresult of the decimation falls within a range of the moving speed inwhich the image quality score evaluated based on the data indicating thecorrespondence between the moving speed of a subject and the imagequality score for data obtained as a result of decimation in the fixedSPP mode is equal to or higher than a predetermined threshold value T4.

In the motion image data conversion apparatus, the block processing unitmay perform the decimation process in one of following modes (a) to (c)selected based on the data of motion, (a) mode in which only spatialdecimation is performed, (b) mode in which spatial decimation andtemporal decimation are performed, or (c) mode in which only temporaldecimation is performed.

In the motion image data conversion apparatus, the motion detector maydetect a motion vector based on blocks that correspond to each other andthat belong to different frames, and the block processing unit maydetermine the decimation mode based on the motion vector and perform thedecimation process in the determined mode.

According to another embodiment, the invention provides a motion imagereproduction apparatus that reproduces motion image data from convertedmotion image data, including a block expander that receives convertedblock data included in the converted motion image data and dataindicating a conversion mode in which the converted block data wasconverted and that expands the converted block data based on the dataindicating the conversion mode, and a combiner that produces frame databy combining blocks reproduced via the block expansion performed by theblock expander, wherein the block expander reproduces the block byperforming the block expansion process in a mode corresponding to atleast one of a spatial decimation process in a fixed sampling pointposition mode, a spatial decimation process in a sampling point positionshifting mode, and a temporal decimation process, performed in theproduction of the converted motion image data, and the combiner producesthe frame data by combining blocks reproduced via the block expansionprocess.

In the motion image reproduction apparatus, when the blocks reproducedby the block expander are blocks reproduced from blocks subjected to thespatial decimation process in the sampling point position shifting mode,the combiner may shift the positions at which the blocks are laid withframe advance.

In the motion image reproduction apparatus, when the blocks reproducedby the block expander are blocks reproduced from blocks subjected to thespatial decimation process in the sampling point position shifting mode,the combiner may shift the positions at which the blocks are laid withframe advance and may perform a pixel value correction process todetermine a pixel value in a pixel gap created when blocks were laid ordetermine a pixel value for an overlapping pixel.

According to another embodiment, the invention provides a method ofconverting motion image data, including the steps of dividing each frameof the motion image data into blocks, detecting the amount of motion ofa subject for each block generated in the block dividing step, andprocessing block data by receiving block data of blocks generated in theblock dividing step and data indicating the amount of motion detected inthe step of detecting the amount of motion, and decimating the blockdata, the block data processing step including the steps of determininga decimation mode as to whether to perform spatial decimation in a fixedsampling point position mode or a sampling point position shifting mode,in accordance with the data of motion, and executing spatial decimationin the fixed SPP mode or the SPP shifting mode in accordance with adetermination made in the decimation mode determination step.

In the method of converting motion image data, the decimation modedetermination step may further include the step of determining adirection in which the sample point position is shifted with frameadvance in the spatial decimation process in the SPP shifting mode, andin the decimation execution step, spatial decimation may be performedwhile shifting the sampling point position in the direction determinedin the decimation mode determination step.

In the method of converting motion image data, in the decimation modedetermination step, the decimation mode may be determined in accordancewith data indicating the correspondence between an image quality scoreand a moving speed of a subject, the image quality score being estimatedbased on data obtained as a result of the decimation process in thefixed sampling point position mode, the determination being made suchthat when a moving speed of a subject indicated by the data of motiondetected in the step of detecting the amount of motion corresponds to anevaluated image quality score equal to or higher than a predeterminedthreshold value, it is determined that the decimation process should beperformed in the fixed process point position mode, but when a movingspeed of a subject indicated by the data of motion detected in the stepof detecting the amount of motion corresponds to an evaluated imagequality score lower than the predetermined threshold value, it isdetermined that the decimation process should be performed in theprocess point position shifting mode. According to another embodiment,the invention provides a method of reproducing motion image data fromconverted motion image data, including the steps of: expanding a blockby receiving converted block data included in the converted motion imagedata and data indicating a conversion mode in which the converted blockdata was converted, and expanding the converted block data based on thedata indicating the conversion mode, and combining blocks reproduced inthe block expansion step to produce frame data, wherein the blockexpansion step reproduces the block by performing the block expansionprocess in a mode corresponding to at least one of a spatial decimationprocess in a fixed sampling point position mode, a spatial decimationprocess in a sampling point position shifting mode, and a temporaldecimation process, performed in the production of the converted motionimage data, and the block combining step produces the frame data bycombining blocks reproduced via the block expansion process.

In the method of reproducing motion image data, in the block combiningstep, when the blocks reproduced by the block expander are blocksreproduced from blocks subjected to the spatial decimation process inthe sampling point position shifting mode, the positions at which to laythe blocks may be shifted with frame advance.

In the method of reproducing motion image data, the block combining stepmay include the step of, when the blocks reproduced by the blockexpander are blocks reproduced from blocks subjected to the spatialdecimation process in the sampling point position shifting mode,shifting the positions at which the blocks are laid with frame advanceand performing a pixel value correction process to determine a pixelvalue in a pixel gap created when blocks were laid or determine a pixelvalue for an overlapping pixel.

According to another embodiment, the invention provides a computerprogram for executing a motion image data conversion process, theprocess including the steps of dividing each frame of the motion imagedata into blocks, detecting the amount of motion of a subject for eachblock generated in the block dividing step, processing block data byreceiving block data of blocks generated in the block dividing step anddata indicating the amount of motion detected in the step of detectingthe amount of motion, and decimating the block data, the block dataprocessing step including the steps of determining a decimation mode asto whether to perform spatial decimation in a fixed sampling pointposition mode or a sampling point position shifting mode, in accordancewith the data of motion; and executing spatial decimation in the fixedSPP mode or the SPP shifting mode in accordance with a determinationmade in the decimation mode determination step.

According to another embodiment, the invention provides a computerprogram for executing a process of reproducing motion image data fromconverted motion image data, the process including the steps ofexpanding a block by receiving converted block data included in theconverted motion image data and data indicating a conversion mode inwhich the converted block data was converted, and expanding theconverted block data based on the data indicating the conversion mode,and combining blocks reproduced in the block expansion step to produceframe data, wherein the block expansion step reproduces the block byperforming the block expansion process in a mode corresponding to atleast one of a spatial decimation process in a fixed sampling pointposition mode, a spatial decimation process in a sampling point positionshifting mode, and a temporal decimation process, performed in theproduction of the converted motion image data, and the block combiningstep produces the frame data by combining blocks reproduced via theblock expansion process.

The computer program may be provided to a general-purpose computersystem capable of executing various computer program codes via a storagemedium such as a CD, an FD, or an MO on which the program is stored in acomputer-readable manner or via a communication medium such as anetwork. By providing the program in the computer-readable form asdescribed above, it becomes possible to execute processes on thecomputer system in accordance with the program.

Note that in the present description, the term “system” is used todescribe a logical collection of a plurality of devices, and it is notnecessarily required that the plurality of devices be disposed in asingle case.

The present invention provides great advantages. That is, in anembodiment, an optimum compression mode is determined depending on afeature of a region of an image, in particular, depending on motion of asubject, and data conversion is performed for each region in thedetermined optimum mode. This makes it possible to compress image dataand decompress the compressed image data with substantially nodegradation in image quality.

In an embodiment, the amount of motion of a subject in a block of motionimage data is detected, and spatial decimation is performed in the fixedSPP mode or the SPP shifting mode depending on the detected amount ofmotion. The spatial decimation in the SPP shifting mode allows thesubject to have a virtual moving speed that allows a super resolutioneffect to occur, thereby achieving data conversion without causingsignificant degradation in image quality.

In an embodiment, the decimation mode is determined in accordance withdata indicating the correspondence between an image quality score and amoving speed of a subject, the image quality score being estimated basedon data obtained as a result of the decimation process in the fixedsampling point position mode, the determination being made such thatwhen a moving speed of a subject in a block of interest indicated by thedata of motion detected by the motion detector corresponds to anevaluated image quality score equal to or higher than a predeterminedthreshold value, it is determined that the decimation process should beperformed in the fixed process point position mode, but when a movingspeed of a subject indicated by the data of motion detected by themotion detector corresponds to an evaluated image quality score lowerthan the predetermined threshold value, it is determined that thedecimation process should be performed in the process point positionshifting mode, thereby making it possible for the subject moving at aspeed that may vary over a wide range to have a virtual speed in a rangein which the super resolution effect can be obtained, and thus achievingdata conversion without causing significant degradation in imagequality.

In an embodiment, blocks are reproduced by performing the blockexpansion process in a mode corresponding to at least one of a spatialdecimation process in the fixed sampling point position mode, a spatialdecimation process in the sampling point position shifting mode, and atemporal decimation process, performed in the production of theconverted motion image data, and the frame data is reproduced bycombining the blocks reproduced via the block expansion process. Thatis, when motion image data compressed by the spatial decimation in theSPP shifting mode is given, motion image data is reproduced byperforming decompression on blocks in a mode selected depending on themode in which original blocks were spatially decimated, thereby makingit possible to reproduce motion image data with high image qualitysimilar to that of the original motion image data.

Further features and advantages of the present invention will becomeapparent from the following description of exemplary embodiments withreference to the attached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing a basic structure of a motion imagedata conversion apparatus that performs data conversion using a superresolution effect;

FIG. 2 is a block diagram showing details of the basic structure of themotion image data conversion apparatus that performs data conversionusing the super resolution effect;

FIG. 3 is a diagram showing a process performed by a block processingunit of a motion image data conversion apparatus;

FIG. 4 is a diagram showing a process performed by a block processingunit of a motion image data conversion apparatus;

FIG. 5 is a diagram showing a process performed by a block processingunit of a motion image data conversion apparatus;

FIG. 6 is a diagram showing a process performed by a block processingunit of a motion image data conversion apparatus;

FIG. 7 is a diagram showing a process performed by a block processingunit of a motion image data conversion apparatus;

FIG. 8 is a diagram showing a process performed by a block processingunit of a motion image data conversion apparatus;

FIG. 9 is a diagram showing a process performed by a block processingunit of a motion image data conversion apparatus;

FIG. 10 is a diagram showing a process performed by a block processingunit of a motion image data conversion apparatus;

FIG. 11 is a diagram showing a process performed by a block processingunit of a motion image data conversion apparatus;

FIG. 12 is a diagram showing a process performed by a block processingunit of a motion image data conversion apparatus;

FIG. 13 is a block diagram showing a structure of a motion image dataconversion apparatus that performs data conversion using the superresolution effect;

FIG. 14 is a block diagram showing a detailed structure of a motionimage data conversion apparatus according to an embodiment of theinvention;

FIG. 15 is a diagram showing a process performed by a block processingunit of a motion image data conversion apparatus;

FIG. 16 is a diagram showing a process performed by a block processingunit of a motion image data conversion apparatus;

FIG. 17 is a diagram showing a process performed by a block processingunit of a motion image data conversion apparatus;

FIG. 18 is a diagram showing a detailed structure of a block processingunit of a motion image data conversion apparatus;

FIG. 19A is diagram showing a decimation process in a fixed samplingpoint position mode, and FIG. 19B is a diagram showing a decimationprocess in a sampling point position shifting mode, performed by a blockprocessing unit of a motion image data conversion apparatus;

FIG. 20 is a graph showing the correspondence between the moving speedof a subject and the image quality of an image reproduced from dataproduced by a block processing unit of a motion image data conversionapparatus by performing a decimation process in the sampling pointposition shifting mode;

FIG. 21 is a graph showing a virtual change in the moving speedresulting from a decimation process in the sampling point positionshifting mode performed by a block processing unit of a motion imagedata conversion apparatus;

FIG. 22 is a graph showing a virtual change in the moving speedresulting from a decimation process in the sampling point positionshifting mode performed by a block processing unit of a motion imagedata conversion apparatus;

FIG. 23 is a table showing the correspondence between the moving speedand the mode of the decimation process performed by a block processingunit of a motion image data conversion apparatus;

FIG. 24 is a diagram showing a process performed by a block processingunit of a motion image data conversion apparatus;

FIG. 25 is a diagram showing a process performed by a block processingunit of a motion image data conversion apparatus;

FIG. 26 is a diagram showing a process performed by a block processingunit of a motion image data conversion apparatus;

FIGS. 27A and 27B are tables showing the correspondence between themoving speed and the mode of the decimation process performed by a blockprocessing unit of a motion image data conversion apparatus;

FIG. 28 is a diagram showing a structure of a block processing unit of amotion image data conversion apparatus, the block processing unitincluding a memory;

FIGS. 29A and 29B are tables showing a process performed by a decimationmode determination unit;

FIG. 30 is a graph showing a manner in which margins of limits ofallowable ranges of the moving speed are defined by introducing anotherthreshold value of the image quality score in addition to a standardthreshold value;

FIG. 31 is a diagram showing a process performed by a block processingunit of a motion image data conversion apparatus;

FIG. 32 is a diagram showing a process performed by a block processingunit of a motion image data conversion apparatus;

FIG. 33 is a diagram showing a process performed by a block processingunit of a motion image data conversion apparatus;

FIG. 34 is a diagram showing a process performed by a block processingunit of a motion image data conversion apparatus;

FIG. 35 is a diagram showing a process performed by a block processingunit of a motion image data conversion apparatus;

FIG. 36 is a diagram showing a process performed by a block processingunit of a motion image data conversion apparatus;

FIG. 37 is a diagram showing a process performed by a block processingunit of a motion image data conversion apparatus;

FIG. 38 is a graph showing the correspondence between the moving speedof a subject and the image quality of an image reproduced from dataproduced by a block processing unit of a motion image data conversionapparatus by performing a decimation process in the sampling pointposition shifting mode;

FIG. 39 is a graph showing a virtual change in the moving speedresulting from a decimation process in the sampling point positionshifting mode performed by a block processing unit of a motion imagedata conversion apparatus;

FIG. 40 is a graph showing a virtual change in the moving speedresulting from a decimation process in the sampling point positionshifting mode performed by a block processing unit of a motion imagedata conversion apparatus;

FIGS. 41A and 41B are tables showing the correspondence between themoving speed and the mode of the decimation process performed by a blockprocessing unit of a motion image data conversion apparatus;

FIG. 42 is a flow chart of a process performed by a motion image dataconversion apparatus;

FIG. 43 is a block diagram showing a structure of a motion imagereproduction apparatus;

FIGS. 44A and 44B are diagrams showing processes performed by a blockdecompressor of a motion image reproduction apparatus;

FIG. 45 is a diagram showing a process performed by a block decompressorof a motion image reproduction apparatus;

FIG. 46 is a diagram showing a process performed by a block decompressorof a motion image reproduction apparatus;

FIGS. 47A and 47B are diagrams showing processes performed by a blockdecompressor of a motion image reproduction apparatus;

FIG. 48 is a diagram showing a block laying process performed by amotion image reproduction apparatus;

FIGS. 49A to 49D are diagrams showing a block laying process performedby a motion image reproduction apparatus;

FIGS. 50A to 50D are diagrams showing a block laying process performedby a motion image reproduction apparatus;

FIGS. 51A to 51D are diagrams showing a block laying process performedby a motion image reproduction apparatus; and

FIGS. 52A and 52B are diagrams showing a block laying process and acorrection process performed by a motion image reproduction apparatus.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

The method and apparatus for converting motion image data, the methodand apparatus for reproducing motion image data, the computer programfor converting motion image data, and the computer program forreproducing motion image data are described below with reference tospecific embodiments in conjunction with the accompanying drawings. Thedescription will be provided in the following order.

(1) Basic structure of a motion image data conversion apparatus usingthe super resolution effect

(2) Structure of a motion image data conversion apparatus that performsa decimation process in an improved manner

(3) Apparatus and method of reproducing motion image data

(1) Basic Structure of a Motion Image Data Conversion Apparatus usingthe Super Resolution Effect

First, described is the basic structure of a motion image dataconversion apparatus that compresses motion image data using the superresolution effect on which the present invention is based. The detailsof the basic structure are disclosed in Japanese Patent Application No.

2003-412501 filed by the present applicant. The motion image dataconversion apparatus is configured to divide an image into small blocksand adaptively decimate pixels or frames depending on the moving speedof respective blocks thereby achieving compression of the motion imagedata.

FIG. 1 shows an example of a structure of the motion image dataconversion apparatus 10. This structure is disclosed in Japanese PatentApplication No. 2003-412501. This motion image data conversion apparatus10 is capable of converting motion image data using the super resolutioneffect such that the data size is reduced without causing degradation inimage quality perceptible by viewers.

The super resolution effect refers to a visual effect based on thenature of visual sense that causes a viewer to perceive the sum of aplurality of images given in a particular period. By nature of humanvisual sense, if human visual sense is stimulated, the stimulation ismemorized for a particular period even after the stimulation is removed(this phenomenon is known as sensory memory). According to many reports,the sensory memory is retained for a period of 10 ms to 200 ms. Thisphenomenon is also called “iconim memory” or “visual informationstorage”. A more detailed description thereof may be found, for example,in “Visual Information Handbook” (edited by the Vision Society of Japan,pp. 229-230). The super resolution effect occurs as a result of acomplicated mixture of functions of human visual sense, in particular, afunction of temporally integrating visual stimulation and a function ofmemorizing stimulation.

The motion image data conversion apparatus 10 shown in FIG. 1 isconfigured to perform motion image data conversion using the superresolution effect due to the human visual function of temporarilyintegrating visual stimulation, so that data is compressed without imagequality degradation perceptible by viewers. The structure of the motionimage data conversion apparatus 10 shown in FIG. 1 is described below.

A block divider 11 divides each frame of an input motion image intoblocks each including a predetermined number of pixels and supplies theresultant blocks to a motion detector 12. The motion detector 12 detectsthe amount of motion of each block supplied from the block divider 11and transmits data indicating the amount of motion together with theblock to a block processor 13. The block processor 13 reduces data sizesof the blocks supplied from the motion detector 12 by performing amotion image conversion process (a compression process) on the blockdepending on the amount of motion. The block processor 13 suppliesresultant block data with the reduced data size to an output unit 14.The output unit 14 combines data of respective blocks with reduced datasizes supplied from the block processor 13 and outputs the data in theform of stream data.

Referring to FIG. 2, details of the respective parts of the motion imagedata conversion apparatus 10 are described below. Each frame of themotion image input to the motion image data conversion apparatus 10 issupplied to an image storage unit 21 of the block divider 11. The imagestorage unit 21 stores the supplied frames. Each time N frames have beenstored in the image storage unit 21 (where N is an integer), the imagestorage unit 21 supplies the N frames to a block dividing unit 22 andalso supplies an M-th (M-thly stored) frame of the N frames to a motiondetecting unit 31 of the motion detector 12. For example, N=4.

The block dividing unit 22 divides each of the N successive framessupplied from the image storage unit 21 into blocks with a predeterminedsize (for example, 8×8 pixels or 16×16 pixels) and outputs the blocks toa block distributor 32 of the motion detector 12. The block dividingunit 22 also supplies a P-th (P-thly stored) frame of the N framesstored in the image storage unit 21 to the motion detecting unit 31 ofthe motion detector 12. Note that the P-the frame is different from theM-th frame.

Next, the details of the motion detector 12 are described below. Themotion detecting unit 31 of the motion detector 12 detects the motionvector of each block of the P-th frame supplied from the block dividingunit 22 of the block divider 11 by means of, for example, interframeblock matching with respect to the M-th frame supplied from the imagestorage unit 21. The detected motion vector is supplied to the blockdistributor 32. The motion vector represents the amount of motionbetween frames in the horizontal direction (along the X axis) and thevertical direction (along the Y axis). To improve the accuracy of thedetection of motion, the motion detecting unit 31 may enlarge the imageand may detect the motion on the enlarged image.

The block distributor 32 of the motion detector 12 receives N blocks(located at the same position of respective of N frames) at a time fromthe block dividing unit 22 and also receives the data indicating themotion of the block of the P-th frame, of the received N blocks, fromthe motion detecting unit 31. The block distributor 32 transfers thereceived N blocks and the data indicating the motion to a blockprocessing unit, which performs processing in a mode determineddepending on the amount of motion, of block processing units 51 to 53 ofthe block processor 13.

More specifically, when the data received from the motion detecting unit31 indicates that the motion in the horizontal (X) direction or thevertical (Y) direction per frame is equal to or greater than 2 pixels,the block distributor 32 supplies the N blocks received from the blockdividing unit 22 and the motion data received from the motion detectingunit 31 to the block processing unit 51. In a case in which the motionper frame is less than 2 pixels but equal to or greater than 1 pixel inboth horizontal and vertical directions, the block distributor 32supplies the N blocks and the motion data to the block processing unit53. When the motion has any other value, the block distributor 32supplies the N blocks and the motion data to the block processing unit52.

That is, the block distributor 32 determines an optimum frame rate andan optimum spatial resolution depending on the amount of motionindicated by the data supplied from the motion detector 12, anddistributes the block image data to the block processing units 51 to 53depending on the frame rate and the spatial resolution.

The criterion for determining the block processing unit to which tosupply the block image data is not limited to that employed above, butthe block processing unit may be determined based on another criterion.

Now, the details of the block processor 13 are described below. Asdescribed above, the block processor 13 includes three block processingunits 51 to 53. The block processing unit 51 performs pixel decimation(spatial decimation) on a total of N blocks (having motion equal to orgreater than 2 pixels/frame in the horizontal or vertical direction)located at the same position of respective N frames supplied from theblock distributor 32 of the motion detector 12, depending on the amountof motion indicated by the data received from the block distributor 32.

More specifically, In the case in which each block includes 8×8 pixels,when the amount of motion in the horizontal direction is equal to orgreater than 2 pixels/frame, the block processing unit 51 divides eachblock into pixel sets each including 1×4 pixels as shown in FIG. 3.Furthermore, as shown in FIG. 4, the block processing unit 51 decimatespixel values p1 to p4 of each 1×4 pixel set into one pixel value equalto one of the pixel values p1 to p4 (p1, in the example shown in FIG. 4)(that is, the block processing unit 51 decimates pixels by a factor of4).

When the amount of motion in the vertical direction is equal to orgreater than 2 pixels/frame, the block processing unit 51 divides eachblock into pixel sets each including 4×1 pixels as shown in FIG. 5, anddecimates pixel values p1 to p4 of each 4×1 pixel set into one pixelvalue equal to one of the pixel values p1 to p4 (p1, in the exampleshown in FIG. 6) as shown in FIG. 6.

When the amount of motion is equal to or greater than 2 pixels/frame inboth vertical and horizontal directions, the block processing unit 51divides each block into pixel sets each including 2×2 pixels as shown inFIG. 7, and decimates pixel values p1 to p4 of each 2×2 pixel set intoone pixel value equal to one of the pixel values p1 to p4 (p1, in theexample shown in FIG. 8) as shown in FIG. 8.

The block processing unit 51 performs the spatial decimation on each offour supplied blocks in the above-described manner. Because data of each4 adjacent pixels is reduced to data of 1 pixel, the data size of eachblock is reduced by a factor of 4, and the total data size of 4 blocksis reduced by a factor of 4. The resultant data of 4 blocks with thedata size reduced by the factor of 4 is supplied from the blockprocessing unit 51 to the output unit 14.

Although in the example shown in FIG. 4, pixel values of each 1×4 pixelset are decimated into one pixel value equal to the pixel value p1 atthe leftmost position, in the example shown in FIG. 6, pixel values ofeach 4×1 pixel set are decimated into one pixel value equal to the pixelvalue p1 located at the top, and in the example shown in FIG. 8, pixelvalues of each 2×2 pixel set are decimated into one pixel value equal tothe pixel value p1 located in the upper left corner, the decimated pixelvalue may be equal to any one of pixel values p1 to p4. Instead of usingone of the pixel values p1 to p4, the pixel value may be calculated fromthe pixel values p1 to p4. For example, the average value of the pixelvalues p1 to p4 may be used.

Next, the operation performed by the block processing unit 52 shown inFIG. 2 is described below. The block processing unit 52 shown in FIG. 2performs frame decimation (temporal decimation) on a total of N blocks(having motion less than 1 pixel/frame in both horizontal and verticaldirections) located at the same position of respective N frames suppliedfrom the block distributor 32 of the motion detector 12.

More specifically, as shown in FIG. 9, the block processing unit 52performs decimation such that four blocks Bi at the same position ofrespective four successive frames F1 to F4 are decimated into one block(block Bi of frame F1 in the example shown in FIG. 9) selected fromthese four blocks. The resultant data of the four blocks whose totaldata size was reduced to ¼ of the original total data size via thetemporal decimation (that is, the data of one block) is supplied fromthe block processing unit 52 to the output unit 14.

The block processing unit 53 performs pixel decimation (spatialdecimation) and frame decimation (temporal decimation) on a total of Nblocks (having motion equal to or greater than 1 pixel/frame but lessthan 2 pixels/frame in both horizontal and vertical directions) locatedat the same position of respective N frames supplied from the blockdistributor 32 of the motion detector 12.

In the decimation process performed by the block processing unit 53,unlike the decimation process performed by the block processing unit 51,each pixel set including pixels p1 to p4 is decimated into a pixel setincluding two pixels (p1 and p3, in the example shown in FIGS. 10 and11) selected from the pixels p1 to p4 as shown in FIGS. 10 and 11.

In the decimation process performed by the block processing unit 51, asdescribed above, the decimation is performed in one of three ways:decimation of 1×4 pixels into one pixel; decimation of 4×1 pixels intoone pixel; and decimation of 2×2 pixels into one pixel. In contrast, inthe decimation process performed by the block processing unit 53, thedecimation is performed in one of two ways: decimation of 1×4 pixelsinto 1×2 pixels; and decimation of 4×1 pixels into 2×1 pixels.

In the frame decimation process performed by the block processing unit53, unlike the frame decimation process performed by the blockprocessing unit 52, arbitrary two blocks (blocks of frames F1 and F3, inthe example shown in FIG. 12) are selected from a total of four blocksBi located at the same position of respective four successive frames F1to F4, and the four blocks are decimated into two selected blocksthereby decimating frames (by a factor of 2).

As a result of the decimation performed by the block processing unit 53on the supplied four blocks, the data size is reduced by a factor of 2via the spatial decimation and also reduced by a factor of 2 via thetemporal decimation, and thus, the total data size of four blocks isreduced to ¼ (=½×½) of the original data size. The resultant data of 4blocks with the data size reduced by the factor of 4 is supplied fromthe block processing unit 53 to the output unit 14.

As described above, the motion image data conversion apparatus 10 shownin FIG. 1 converts (compresses) an input motion image into a motionimage with a reduced data size. Note that the data conversion isperformed using the super resolution effect based on the nature of humanvisual sense so that a viewer does not perceive degradation in imagequality.

More specifically, the block distributor 32 determines the optimum framerate and the optimum spatial resolution depending on the amount ofmotion informed by the motion detector 12, the block distributor 32determines which one of the block processing units 51 to 53 shouldperform image data conversion, based on the determined optimum framerate and spatial resolution, and the block distributor 32 supplies imagedata to the selected block processing unit. That is, it is possible toperform data conversion in different modes using the block processingunits 51 to 53 so that the motion image data is converted without imagequality degradation perceptible by viewers. As described above, thesuper resolution effect refers to a visual effect based on the nature ofvisual sense that causes a viewer to perceive the sum of a plurality ofimages given in a particular period. The super resolution effect occursas a result of a complicated mixture of functions of human visual sense,in particular, a function of temporally integrating visual stimulationand a function of memorizing stimulation. The motion image dataconversion apparatus 10 shown in FIG. 1 is configured to perform motionimage data conversion using the super resolution effect due to the humanvisual function of temporarily integrating visual stimulation.

A more detailed explanation of the characteristics of human visual senseand the super resolution effects may be found, for example, in JapanesePatent Application No. 2003-412501. As described in Japanese PatentApplication No. 2003-412501, the super resolution effect occurs when thefollowing conditions are satisfied.

To obtain the super resolution effect when pixel decimation is performedby a factor of m, it is required to substantially cancel out thealiasing components of first to (m−1)-th orders caused by thedecimation. When the following two equations (1) and (2) are satisfied,the aliasing components of k-th (k=1, 2, . . . , (m-1)) order arecancelled out.

$\begin{matrix}{{\sum\limits_{t}{\cos \left( {2\pi \; k\; \varphi_{t}} \right)}} = 0} & {{Equation}\mspace{14mu} (1)} \\{{\sum\limits_{t}{\sin \left( {2\pi \; k\; \varphi_{t}} \right)}} = 0} & {{Equation}\mspace{14mu} (2)}\end{matrix}$

where φ_(t) is the sampling position deviation that occurs when pixelsare decimated, and that is defined by the following equation (3):

$\begin{matrix}{\varphi_{t} = {{- \frac{v}{m}}\frac{t}{T}}} & {{Equation}\mspace{14mu} (3)}\end{matrix}$

Note that in the above equation, when the sampling position is shiftedto the right, the φ_(t) has a positive value (unlike the definition inJapanese Patent Application No. 2003-412501).

If the decimation factor m and the amount of local motion v satisfyequations (1) and (2), the super resolution effect occurs and thusdegradation in image quality is not easily perceived by viewers.

In the motion image data conversion apparatus 10 described above withreference to FIGS. 1 to 12, pixel decimation by a factor of 4 isperformed when the local motion per frame is equal to or greater than 2pixels in the horizontal or vertical direction. However, when the motionper frame is equal to 4 pixels, that is, when the amount of local motionv=4, if the pixel decimation factor m=4, then the sampling positiondeviation φ_(t) given by equation (3) can take an integer greater than0.

That is, equation (1) is not satisfied for all values of k, and thus thesuper resolution effect does not occur. Similarly, when the pixeldecimation factor m=2 and the amount of local motion v=1, or when thepixel decimation factor m=4 and the amount of local motion v=2, equation(1) is also not satisfied for all values of k, and thus the superresolution effect does not occur and significant degradation in imagequality occurs. That is, in the motion image data conversion apparatus10 described above with reference to FIGS. 1 to 12, although when thepixel decimation factor m and the amount of local motion v satisfyparticular conditions, the super resolution can be achieved anddegradation in image quality due to the reduction in the data size isnot perceived by viewers, the super resolution cannot be achieved andsignificant degradation in image quality occurs when the pixeldecimation factor m and the amount of local motion v do not satisfy theparticular conditions. In an embodiment of the present invention, theabove problem is solved as follows.

(2) Structure of a Motion Image Data Conversion Apparatus that Performsa Decimation Process in an Improved Manner

In an embodiment described below, a motion image data conversionapparatus is capable of performing a decimation process in an improvedmanner such that the super resolution effect can be achieved more easilyunder much wider conditions and thus the data conversion (datacompression) can be performed without causing significant degradation inimage quality.

The motion image data conversion apparatus 100 according to the presentembodiment is described below with reference to FIG. 13. The motionimage data conversion apparatus 100 shown in FIG. 13 is capable of, aswith the motion image data conversion apparatus 10 described above withreference to FIG. 1, reducing the data size using the super resolutioneffect based on the nature of human visual sense such that degradationin image quality due to the reduction in the data size is not perceivedby viewers. Note that in the motion image data conversion apparatus 100according to the present embodiment, the super resolution effect occursregardless of whether the pixel decimation factor m and the amount oflocal motion v satisfy particular conditions, and thus data sizereduction does not cause perceptible degradation in image quality.

Referring to FIG. 13, the structure of the motion image data conversionapparatus 100 is described below. A block divider 110 divides each frameof input image data into blocks and supplies the resultant blocks to amotion detector 120. The motion detector 120 detects the amount ofmotion of each block supplied from the block divider 110 and transmitsdata indicating the amount of motion together with the block to a blockprocessor 130. The block processor 130 reduces data sizes of the blockssupplied from the motion detector 120 by performing a motion imageconversion process on the blocks depending on the amount of motion. Theblock processor 130 supplies resultant block data with reduced datasizes to an output unit 140. The output unit 140 combines data ofrespective blocks with reduced data sizes supplied from the blockprocessor 130 and outputs the data in the form of stream data.

Referring to FIG. 14, details of the respective parts of the motionimage data conversion apparatus 100 are described below. First, theblock divider 110 is described. Each frame of the motion image input tothe motion image data conversion apparatus 100 is supplied to an imagestorage unit 111 of the block divider 110. The image storage unit 111stores the supplied frames. Each time N frames have been stored in theimage storage unit 111 (where N is an integer), the image storage unit111 supplies the N frames to a block dividing unit 112 and also suppliesan M-th (M-thly stored) frame of the N frames to a motion detecting unit121 of the motion detector 120. For example, N=4.

The block dividing unit 112 divides each of the N successive framessupplied from the image storage unit 111 into blocks with apredetermined size (for example, 8×8 pixels or 16×16 pixels) and outputsthe blocks to a block distributor 122 of the motion detector 120. Theblock dividing unit 112 also supplies a P-th (P-thly stored) frame ofthe N frames stored in the image storage unit 111 to the motiondetecting unit 121 of the motion detector 120. Note that the P-the frameis different from the M-th frame.

Now, the details of the motion detector 120 are described below. Themotion detecting unit 121 of the motion detector 120 detects the motionvector of each block of the P-th frame supplied from the block dividingunit 112 of the block divider 110 by means of, for example, interframeblock matching with respect to the M-th frame supplied from the imagestorage unit 111. The detected motion vector is supplied to the blockdistributor 122 and the block processing units 131 and 132. The motionvector represents the amount of motion between frames in the horizontaldirection (along the X axis) and the vertical direction (along the Yaxis). For example, when M=2 and P=3, the motion vector indicates theamount of motion per frame in the horizontal (X) direction and vertical(Y) direction. The method of detecting the motion vector is not limitedto the block matching, but the motion vector may be detected using othermethods.

The block distributor 122 of the motion detector 120 receives N blocks(located at the same position of respective of N frames) at a time fromthe block dividing unit 112 and also receives the data indicating themotion of the block of the P-th frame of the received N blocks from themotion detecting unit 121. The block distributor 122 transfers thereceived N blocks to one of block processing units 131 to 133 of theblock processor 130, selected depending on the amount of motion.Hereinafter, the “block processing unit 130” will be used when it is notnecessary to identify a particular one of block processing units 131 to133.

More specifically, when the amount of motion per frame indicated by thedata supplied from the motion detecting unit 121 is greater in one ofhorizontal (X) and vertical directions than in the other direction, if agreater amount of motion is equal to or greater than 2 pixels/frame, theblock distributor 122 supplies the N blocks received from the blockdividing unit 112 to the block processing unit 131. In a case in whichthe greater amount of motion is equal to or greater than 1 pixel/framebut less than 2 pixels/frame, the block distributor 122 supplies the Nblocks to the block processing unit 132. When the amount of motion hasany other value, the block distributor 122 supplies the N blocks to theblock processing unit 133.

That is, depending on the amount of motion, the blocks are supplied toone of block processing units 131 to 133, as follows:

(a) When the amount of motion≧2 pixels/frame, the blocks are supplied tothe block processing unit 131 (to perform spatial decimation).

(b) When 2 pixels/frame>amount of motion≧1 pixel/frame, the blocks aresupplied to the block processing unit 132 (to perform temporal andspatial decimation).

(c) When 1 pixel/frame>amount of motion, the blocks are supplied to theblock processing unit 133 (to perform temporal decimation).

That is, when the amount of motion≧2 pixels/frame, spatial decimation isperformed by the block processing unit 131. When 2 pixels/frame>amountof motion≧1 pixel/frame, temporal and spatial decimation is performed bythe block processing unit 132. When 1 pixel/frame>amount of motion,temporal decimation is performed by the block processing unit 133.

As described above, the block distributor 122 determines an optimumframe rate and an optimum spatial resolution depending on the amount ofmotion indicated by the motion data supplied from the motion detectingunit 121, and distributes the block image data to the block processingunits 131 to 133 depending on the frame rate and the spatial resolution.The criterion for determining the block processing unit to which tosupply the block image data is not limited to that employed above, butthe block processing unit may be determined based on another criterion.

For example, when the amount of motion per frame is greater in one thehorizontal (X) direction and the vertical (Y) direction than in theother direction, if a greater amount of motion is equal to or greaterthan 2 pixels/frame, the N blocks output from the block divider 112 maybe supplied to the block processing unit 131, but when the greateramount of motion has any other value, the N blocks may be supplied tothe block processing unit 133. In this case, the block processing unit132 does not perform any process.

Now, the details of the block processor 130 are described below. In thisembodiment, the block processor 130 includes three block processingunits 131 to 133.

When the amount of motion of N blocks located at the same position ofrespective N successive frames supplied from the block distributor 122of the motion detector 120 is equal to or greater than 2 pixel/frame inthe horizontal or vertical direction, the block processing unit 131performs pixel decimation (spatial decimation) depending on a greateramount of motion of amounts of motion in the horizontal and verticaldirections indicated by the motion data supplied from the motiondetecting unit 131. That is, the block processing unit 131 performsspatial decimation on N blocks having an amount of motion>2pixels/frame.

More specifically, in the case in which each block includes 8×8 pixels,when the amount of motion in the horizontal direction is equal to orgreater than 2 pixels/frame, the block processing unit 131 divides eachblock into pixel sets each including 1×4 pixels as shown in FIG. 15A.

Furthermore, the block processing unit 131 decimates pixel values p1 top4 of each 1×4 pixel set into one pixel equal to one of the pixel valuesp1 to p4 (that is, four pixels are decimated into one pixel, in otherwords, pixels are decimated by a factor of 4) in one of modes shown inFIGS. 15B, 16, and 17.

In the decimation mode shown in FIG. 15B, decimation is performed in asimilar manner to the decimation process (shown in FIG. 4) performed bythe motion image data conversion apparatus having the basic structure.That is, in this decimation mode, for all N successive frames (k-th to(k+3)-th frames), four pixel values are decimated into onerepresentative pixel value at a sampling point (p1 in the example shownin FIG. 15B).

In the decimation modes shown in FIGS. 16 and 17, rather than employinga pixel value at the same position as a sampling point for all Nsuccessive frames (k-th to (k+3)-th frames), the sampling point positionof the representative pixel value to which four pixel values aredecimated is varied frame by frame.

More specifically, in the decimation mode shown in FIG. 16, the samplingpoint position of the representative pixel value, to which four pixelvalues p1 to p4 which are in a horizontal line and which are located atthe same positions for all four frames (k-th to (k+3)-th frames) aredecimated, is shifted by one pixel to the right, frame by frame, asfollows:

p1 is selected as the sampling point for the k-th frame;

p2 is selected as the sampling point for the (k+1)-th frame;

p3 is selected as the sampling point for the (k+2)-th frame; and

p4 is selected as the sampling point for the (k+3)-th frame.

In the decimation mode shown in FIG. 17, the sampling point position ofthe representative pixel value, to which four pixel values p1 to p4which are in a horizontal line and which are located at the samepositions for all four frames (k-th to (k+3)-th frames) are decimated,is shifted by one pixel to the left, frame by frame, as follows:

p4 is selected as the sampling point for the k-th frame;

p3 is selected as the sampling point for the (k+1)-th frame;

p2 is selected as the sampling point for the (k+2)-th frame; and

p1 is selected as the sampling point for the (k+3)-th frame.

As described above, in the decimation modes shown in FIGS. 16 and 17, inN successive frames (k-th to (k+3)-th frames), four pixel values aredecimated into one pixel value at the sampling point position which isshifted frame by frame rather than is fixed.

The block processing unit 131 selects one of decimation modes shown inFIGS. 15 to 17 depending on motion indicated by the motion data suppliedfrom the motion detecting unit 121 of the motion detector 120.

The details of the block processing unit 131 are described below withreference to FIG. 18. The block processing unit 131 includes adecimation mode determination unit 151 and a decimation execution unit152. The motion data indicating the amount of motion output from themotion detecting unit 121 is applied to the decimation modedetermination unit 151. As described above, blocks supplied to the blockprocessing unit 131 are those blocks whose motion>2 pixels/frame, andthus the amount of motion indicated by the motion data input to thedecimation mode determination unit 151 of the block processing unit 131is equal to or greater than 2 pixels/frame.

Based on the amount of motion, the decimation mode determination unit151 selects one of the modes shown in FIGS. 15 to 17 in which to performdecimation. The details of the manner in which to select the decimationmode will be described later. In accordance with the determination madeby the decimation mode determination unit 151, the decimation executionunit 152 executes the decimation process in one the modes shown in FIGS.15 to 17.

In the decimation process, by shifting the sampling point positiondepending on the frame as shown in FIG. 16 or 17, the following effectsare obtained. Hereinafter, the decimation mode in which the samplingpoint position is shifted depending on the frame will be referred to asa “sampling point position shifting mode” (SPP shifting mode) or a“decimation position shifting mode”. Note that the SPP shifting mode hastwo submodes, in one of which the sampling point position is shifted tothe right with frame advance as shown in FIG. 16, and in the other oneof which the sampling point position is shifted to the left with frameadvance as shown in FIG. 17.

Shifting of the decimation position is equivalent to adding 1/m to thesampling position deviation φ_(t) in equation (3) frame by frame (in thecase in which the sampling point position is shifted to the right (FIG.16)) or equivalent to subtracting 1/m from the sampling positiondeviation φ_(t) in equation (3) frame by frame (in the case in which thesampling point position is shifted to the left (FIG. 17)), where m isthe decimation factor (m pixels are decimated into one pixel).

When the decimation position is shifted, the sampling position deviationφ′_(t) is given by equation (4).

$\begin{matrix}\begin{matrix}{\varphi_{t}^{\prime} = {\varphi_{t} \pm {\frac{1}{m}\frac{t}{T}}}} \\{= {{{- \frac{v}{m}}\frac{t}{T}} \pm {\frac{1}{m}\frac{t}{T}}}} \\{= {\frac{- \left( {v \mp 1} \right)}{m}\frac{t}{T}}}\end{matrix} & {{Equation}\mspace{14mu} (4)}\end{matrix}$

From equation (4), it can be seen that, from the point of view of thetheory of signal processing, shifting of the sampling point position inthe same direction as a direction in which a subject is moving isequivalent to deceleration of the moving speed of the subject, andshifting of the sampling point position in a direction opposite to thedirection in which the subject is moving is equivalent to accelerationof the moving speed of the subject. As a matter of course, theacceleration or the deceleration caused by the shifting of the samplingpoint position does not mean that the actual moving speed of the subjectchanges. Specific examples of decimation processes are described belowwith reference to FIGS. 19A and 19B.

In the example shown in FIG. 19A, a subject 201 (represented by fivepixels p1 to p5 of a k-th frame) does not move during a passage of fourframes (k-th to (k+3)-th frames). In the example shown in FIG. 19B, thesubject 201 moves at a speed of v=1 pixel/frame during the passage offour frames (k-th to (k+3)-th frames).

The block processing unit 131 of the motion image data conversionapparatus 100 shown in FIGS. 13 and 14 receives blocks whose motion inthe horizontal direction is equal to or greater than 2 pixels/frame, andthus frame data such as that shown in FIG. 19A or 19B is not input tothe block processing unit 131. On the other hand, if data has motionequal to or greater than 2 pixels/frame in the horizontal direction,decimation is performed in one of modes shown in FIGS. 15 to 17depending on the amount of motion. That is, the decimation position isshifted or not shifted depending on the amount of motion.

Referring to FIGS. 19A and 19B, the decimation process performed whilefixing the sampling point position and the decimation process performedwhile shifting the sampling point position are described below. In theexample shown in FIG. 19A, a subject is at rest, and decimation isperformed in the fixed SPP mode. In the example shown in FIG. 19B, asubject moves at a speed of v=1 pixel/frame, and decimation is performedwhile shifting the sampling point position in a similar manner to theexample described earlier with reference to FIG. 16.

For the data shown in FIG. 19A in which the subject does not move, ifdecimation is performed by a factor of 4, that is, if four pixels aredecimated into one pixel in the fixed SPP (sampling point position) modeas described above with reference to FIG. 15, the pixel value p1 offirst four pixels and the pixel value p5 of next four pixels are output,and no change occurs in position of output pixel values. On the otherhand, for the data shown in FIG. 19B in which the subject moves in thehorizontal direction at the speed of v=1 pixel/frame, if decimation isperformed in the SPP (sampling point position) shifting mode in whichthe sampling point position is shifted frame by frame as described abovewith reference to FIG. 16, the sampling point position, the pixel valueat which is employed as the representative pixel value for four pixelvalues in decimation, is shifted frame by frame as shown in FIG. 19B,that is, the sampling point position is shifted as follows.

p1 and p5 for a k-th frame

p2 and p6 for a (k+1)-th frame

p3 and p7 for a (k+2)-th frame

p4 and p8 for a (k+3)-th frame

As a result, in each frame, the same pixel values are output for thesubject moving at the speed of 1 pixel/frame shown in FIG. 19B and thesubject at rest shown in FIG. 19A. That is, the data which is output asa result of performing decimation on a motion image including a subjectmoving at a speed of v=1 pixel/frame in the SPP shifting mode, in whichthe sampling point position is shifted frame by frame as described abovewith reference to FIG. 16, is identical to the data which is output as aresult of performing decimation on a subject moving at a speed of 0(=v−1) in the fixed SPP mode described above with reference to FIG. 15.That is, performing the decimation process shown in FIG. 16 on a movingsubject is equivalent to performing the decimation process shown in FIG.15 on a subject whose moving speed is reduced by 1.

From the above discussion, it can be seen that it is possible tovirtually accelerate or decelerate the moving speed of a subject. In themotion image data conversion apparatus 10 described earlier withreference to FIGS. 1 to 12, the super resolution effect can be obtainedonly when the decimation factor m and the amount of local motion vsatisfy particular conditions, and significant degradation in imagequality occurs when the decimation factor m and the amount of localmotion v do not satisfy the particular conditions. In contrast, in themotion image data conversion apparatus 100 according to the presentembodiment described above with reference to FIG. 13 and other figures,the block processor 130 performs decimation in an optimum mode selecteddepending on the amount of motion such that the moving speed of asubject is virtually changed to a value at which the super resolutioneffect can be obtained. Thus, the motion image data conversion apparatus100 according to the present embodiment can obtain the super resolutioneffect even for data for which the motion image data conversionapparatus 10 described earlier with reference to FIGS. 1 to 12 cannotobtain a sufficient super resolution effect, and thus the motion imagedata conversion apparatus 100 can perform image data conversion(compression) even for such data without causing significant degradationin image quality.

Now, a further discussion is made below as to when it is difficult toobtain the super resolution effect for a moving subject. FIG. 20 is agraph showing the quality of image data reproduced (decompressed) fromcompressed data produced by performing decimation (compression) by afactor of 4 in the fixed SPP mode on original data of a moving imageincluding a subject moving at a speed of v per frame, as a function ofthe moving speed v of the subject. In FIG. 20, the moving speed v(pixels/frame) of the subject is plotted along the horizontal direction,and the image quality is plotted along the vertical axis. Note that theresult shown in FIG. 20 is based on subjective evaluation on imagequality for various subject speeds, and the result is plotted as animage evaluation curve 205. In FIG. 20, T denotes a threshold value ofthe image quality score. When the score is higher than the thresholdvalue T, the image quality is regarded as good.

When decimation (by a factor of 4) is performed in the fixed SPP mode asdescribed earlier with reference to FIG. 15 for various subjects movingat moving speeds v in the range from P to Q (pixels/frame), the imagequality score higher than the threshold value T was obtained in regionsA shown in FIG. 20. In regions B other than the regions A, the imagequality score was lower than the threshold value T.

In the regions A in which the image quality score was higher than thethreshold value T, the moving speed v of the subject is within the rangefrom a to b, c to d, or greater than e. When the moving speed is withinone of those ranges, high image quality can be obtained even whendecimation is performed in the fixed SPP mode described earlier withreference to FIG. 15. That is, the super resolution effect is obtainedin those regions. On the other hand, when the moving speed v per frameis within one of the ranges B, that is, the moving speed is not withinany one of the ranges a to b, c to d, and greater than e, the imagequality score is lower than the threshold value T. That is, in thoseregions B, the super resolution effect is not obtained or insufficient.

Based on the evaluation result described above, when the moving speed ofa subject is within the regions B shown in FIG. 20, decimation isperformed in the SPP shifting mode such that the moving speed of thesubject is virtually accelerated or decelerated as shown in FIG. 16 or17. As a result, the super resolution effect is obtained as describedabove with reference to FIG. 19, and thus as high image quality as thatobtained in the regions A is obtained.

Regions B-1 and B-2 shown in FIG. 21 are part of the regions B shown inFIG. 20. In these regions B-1 and B-2, the image quality score is lessthan the threshold value, because the moving speed v of a subject doesnot have a value at which a sufficient super resolution effect occurs.When a subject moves at a speed in the range B-1 or B-2, if the subjectis virtually accelerated (by +Δv1 or +Δv2), that is, if decimation isperformed while shifting the decimation position in a direction oppositeto a direction in which the subject is actually moves (for example, whenthe subject is moving to the right, the sampling point position isshifted to the left as shown in FIG. 17), the region B-1 shown in FIG.21 is effectively moved to a region C-1 shown in FIG. 21, and the regionB-2 shown in FIG. 21 is effectively moved to a region C-2 shown in FIG.21. As a result, the virtual moving speed per frame, v, of the subjectfalls within one of regions (C-1 and C-1 shown in FIG. 21) in which theimage quality score can be equal to or greater than the threshold valueT. That is, by performing decimation while shifting the sampling pointposition to the left as the frame advances as shown in FIG. 17, it ispossible to perform image data conversion (compression) without causingsignificant degradation in image quality.

Similarly, when a subject is moving at a speed in a range B-3 shown inFIG. 22, if the subject is virtually decelerated (by −Δv3), that is, ifdecimation is performed while shifting the decimation position in thesame direction as a direction in which the subject is actually moves(for example, when the subject is moving to the right, the decimationprocess is performed as shown in FIG. 16), the region B-3 shown in FIG.22 is effectively moved to a region C-3 shown in FIG. 22. As a result,the virtual moving speed per frame, v, of the subject falls within theregion (C-3 shown in FIG. 22) in which the image quality score can beequal to or greater than the threshold value T. That is, by performingdecimation while shifting the sampling point position to the right asthe frame advances as shown in FIG. 16, it is possible to perform imagedata conversion (compression) without causing significant degradation inimage quality.

In the motion image data conversion apparatus 100 shown in FIGS. 13 and14 according to the present embodiment, the block processing unit 131performs spatial pixel decimation by a factor of 4 only for blocks whosemotion in the horizontal or vertical direction is equal to or greaterthan 2 pixels/frame, and thus blocks whose moving speed v is less than 2pixels/frame in the graphs shown in FIGS. 20 to 22 are not subjected tothe process by the block processing unit 131. However, as describedearlier, the above-described criterion used by the block dividing unit122 to determine which one of the block processing units 131 to 133 ofthe block processor 130 to supply the block to is merely one example,and the block distributor 122 may supply the block whose motion is lessthan 2 pixels/frame to the block processing unit 131 to perform spatialdecimation in one of decimation modes described above with reference toFIGS. 15 to 17. Therefore, the moving speed v less than 2 pixels/frameis also included in the graphs shown in FIGS. 20 to 22.

Note that the image quality evaluation curves shown in FIGS. 20 to 22are merely examples based on a subjective evaluation result, and theimage quality may be evaluated in many ways other than that describedabove and the decimation mode may be determined in accordance with aresult of such an evaluation.

In the block processing unit 131 shown in FIG. 14 responsible forprocessing block data with motion equal to or greater than 2pixels/frame, the decimation mode determination unit 151 shown in FIG.18 determines which one of modes described below should be used based onthe amount of motion indicated by motion data received from the motiondetector, and the decimation execution unit 152 shown in FIG. 18executes decimation in the determined mode.

(a) Decimation is performed while fixing sampling point positions (FIG.15).

(b) Decimation is performed while shifting the sampling point positionto the right as the frame advances (FIG. 16).

(c) Decimation is performed while shifting the sampling point positionto the left as the frame advances (FIG. 17).

When the decimation mode determination unit 151 shown in FIG. 18determines the decimation mode, for example, the image qualityevaluation curves shown in FIGS. 20 to 22 are used. That is, when themoving speed is in a region in which an image quality score equal to orgreater than the threshold value T is obtained judging from the imagequality curve, the mode (a) in which decimation is performed whilefixing the sampling point position (FIG. 15) is used. When the movingspeed is in a region in which the image quality score estimated from theimage quality curve is less than the threshold value T, decimation isperformed in the mode (b) in which decimation is performed whileshifting the sampling point position to the right as the frame advances(FIG. 16) or the mode (c) in which decimation is performed whileshifting the sampling point position to the left as the frame advances(FIG. 17) such that the amount of virtual motion falls within a range inwhich the image quality estimated based on the image quality curve isequal to or greater than the threshold value T.

FIG. 23 is a table showing an example of a manner in which thedecimation mode determination unit 151 shown in FIG. 18 determines thedecimation mode in accordance with the amount of motion indicated bymotion data supplied from the motion detector. In the example shown inFIG. 23, the decimation mode is determined based on the image qualityevaluation curves shown in FIGS. 20 to 22.

In rows (a) and (b) in the table shown in FIG. 23, described is a mannerof decimation performed when a subject is moving to the right and whenthe moving speed v is in a range (a) 2 pixels/frame≦v<c pixels/frame or(b) f pixels/frame≦v<c pixels/frame. In this case, decimation isperformed while shifting the sampling point position to the left as theframe advances (FIG. 17) to virtually accelerate the moving speed. Thisprocess corresponds to the above-described process which causes theregion B-1 shown in FIG. 21 to move into the region C-1 or the regionB-2 to C-2.

In rows (c) and (d) in the table shown in FIG. 23, described is a mannerof decimation performed when a subject is moving to the left and whenthe moving speed v is in a range (c) 2 pixels/frame≦v<c pixels/frame or(d) f pixels/frame≦v<c pixels/frame.

In this case, decimation is performed while shifting the sampling pointposition to the right as the frame advances (FIG. 16) to virtuallyaccelerate the moving speed. This process corresponds to theabove-described process which causes the region B-1 shown in FIG. 21 tomove into the region C-1 or the region B-2 to C-2.

In a row (e) in the table shown in FIG. 23, described is a manner ofdecimation performed when a subject is moving to the right and when themoving speed v is in a range d pixels/frame≦v<f pixels/frame.

In this case, decimation is performed while shifting the sampling pointposition to the right as the frame advances (FIG. 16) to virtuallydecelerate the moving speed. This process corresponds to theabove-described process which causes the region B-3 shown in FIG. 22 tomove into the region C-3.

In a row (f) in the table shown in FIG. 23, described is a manner ofdecimation performed when a subject is moving to the left and when themoving speed v is in a range d pixels/frame≦v<f pixels/frame.

In this case, decimation is performed while shifting the sampling pointposition to the left as the frame advances (FIG. 17) to virtuallydecelerate the moving speed. This process corresponds to theabove-described process which causes the region B-3 shown in FIG. 22 tomove into the region C-3.

In a row (g) in the table shown in FIG. 23, described is a manner ofdecimation performed when a subject is moving to the right or left andwhen the moving speed v has a value other than those described above,that is, the moving speed v is in a range c pixels/frame≦v<dpixels/frame or e pixels/frame≦v.

In this case, rather than in the mode in which sampling point positionis shifted as the frame advances, decimation is performed in the mode inwhich the sampling point position is fixed (FIG. 15). This processcorresponds to the above-described process performed on the blockincluding the subject moving at a speed in the range corresponding tothe region A shown in FIG. 20.

In the example described above, the process is performed on blocks withmotion in a horizontal direction. For blocks with motion in a verticaldirection, the process can be performed in a similar manner. That is, inthe block processing unit 131 shown in FIG. 14, the decimation modedetermination unit 151 shown in FIG. 18 determines which one of modesdescribed below should be used based on the amount of motion indicatedby motion data received from the motion detector, and the decimationexecution unit 152 shown in FIG. 18 executes decimation in thedetermined mode.

(a) Decimation is performed while fixing sampling point positions (FIG.24B).

(b) Decimation is performed while shifting the sampling point positiondownward as the frame advances (FIG. 25).

(c) Decimation is performed while shifting the sampling point positionupward as the frame advances (FIG. 26).

More specifically, in the case in which each block includes 8×8 pixels,when the amount of motion in the vertical direction is equal to orgreater than 2 pixels/frame, the block processing unit 131 divides eachblock into pixel sets each including 1×4 pixels as shown in FIG. 24A.The block processing unit 131 decimates pixel values p1 to p4 of each1×4 pixel set into one pixel value equal to one of the pixel values p1to p4 (that is, four pixels are decimated into one pixel, in otherwords, pixels are decimated by a factor of 4) in one of modes shown inFIGS. 24B, 25, and 26.

When the decimation mode determination unit 151 shown in FIG. 18determines the decimation mode, the determination can be made based onimage quality evaluation curves which are similar to those shown inFIGS. 20 to 22 except that the amount of motion in the verticaldirection is plotted along the horizontal axis. That is, when the movingspeed is in a region in which an image quality score equal to or greaterthan the threshold value T is obtained judging from the image qualitycurve, the mode (a) in which decimation is performed while fixing thesampling point position (FIG. 24) is used. When the moving speed is in aregion in which the image quality score estimated from the image qualitycurve is less than the threshold value T, decimation is performed in themode (b) in which decimation is performed while shifting the samplingpoint position downward as the frame advances (FIG. 25) or the mode (c)in which decimation is performed while shifting the sampling pointposition upward as the frame advances (FIG. 26) such that the amount ofvirtual motion falls within a range in which the image quality estimatedbased on the image quality curve is equal to or greater than thethreshold value T.

FIG. 27A is a table summarizing processing modes that are selected basedon image quality evaluation curves similar to those shown in FIGS. 20 to22 depending on the moving speed v per frame of a subject. FIG. 27B is atable showing the details of the respective modes 1 to 3 shown in FIG.27A.

The processing modes shown in FIG. 27A are described below.

When 2≦v<c, decimation is performed in the processing mode 3 in whichdecimation position is shifted in a direction opposite to a direction inwhich a subject is moving.

When c≦v<d, decimation is performed in the processing mode 1 in whichdecimation position is not shifted.

When d≦v<f, decimation is performed in the processing mode 2 in whichdecimation position is shifted in the same direction as a direction inwhich a subject is moving.

When f≦v<e, decimation is performed in the processing mode 3 in whichdecimation position is shifted in a direction opposite to a direction inwhich a subject is moving.

When e≦v, decimation is performed in the processing mode 1 in whichdecimation position is not shifted.

By switching the decimation mode depending on the moving speed v perframe of a subject included in a block in the above-described manner, itis possible to virtually change the moving speed of the subject suchthat the virtual moving speed falls within the range A shown in FIG. 20,thereby achieving image data conversion so that image quality scoreequal to or higher than the threshold value T is guaranteed.

The block processing unit 131 performs decimation in one of modesdepending on the moving speed v per frame of a subject included in ablock as described above, and outputs resultant data to the output unit140 (FIGS. 13 and 14). In a case in which spatial decimation isperformed by a factor of 4, the block processing unit 131 performs thespatial decimation on each of four supplied blocks (each set of adjacentfour pixels is decimated into one pixel), and thus the data size of eachblock is reduced to ¼ of the original data size, and the total data sizeof four blocks is also reduced to ¼ of the original total data size. Theresultant data of four blocks whose data size was reduced to ¼ of theoriginal data size is supplied from the block processing unit 131 to theoutput unit 14.

Although in the above-described example of the decimation processperformed by the block processing unit 131, the decimation factor m=4and the number of frames N=4, the parameters are not limited to thesevalues. Note that the image quality evaluation curves 205 indicating theimage quality as a function of the moving speed of a subject shown inFIGS. 20 to 22 are for the decimation factor m=4, and the image qualityevaluation curve varies depending on the values of parameters. Thus, itis necessary to determine an image quality evaluation curve for theparameter values employed, and determine the decimation mode dependingon the determined image quality evaluation curve.

In the example of the decimation process in the fixed SPP mode describedabove with reference to FIG. 15, the pixel value p1 at the leftmostposition of each set of 1×4 pixels in a horizontal line is selected. Inthe example of the decimation process in the fixed SPP mode describedabove with reference to FIG. 24, the pixel value p1 at the top of eachset of 4×1 pixels in a vertical line is selected. However, the pixelpositions are not limited to those employed in the above example, but apixel value at another position may be selected. In the example of thedecimation process in the mode in which the sample point position isshifted, described earlier with reference to FIGS. 16 and 17, the pixelsample position is shifted to the right or left starting from a pixel atthe leftmost or rightmost position of a set of 1×4 pixels, shifting maybe started from another position. This also applies to the decimationprocess in which the sample point position is shifted in the verticaldirection, described above with reference to FIGS. 25 and 26.

In the examples described above, the threshold value T is predeterminedas a criterion for evaluation of the image quality. Depending on whetherthe image quality score is higher than the threshold value T, adetermination is made as to whether to perform spatial decimation in thefixed SPP mode or in the SPP shifting mode, and as to how to shift thedecimation position when the decimation is performed in the SPP shiftingmode. Alternatively, the structure of the block processing unit 131shown in FIG. 18 may be modified as shown in FIG. 28, the limits of theallowable ranges of the moving speed may have margins of Δv1 to Δv9, andthe mode of the spatial decimation may be determined depending on whichrange with margins the moving speed falls in, as shown in FIG. 29. Ifthe mode of the spatial decimation performed on a block is switched veryfrequently with passage of time, degradation in image quality can occur.Such degradation in image quality can be avoided by using the structureshown in FIG. 28 and introducing margins (Δv1 to Δv9) in the range ofallowable moving speed. The flow of determining the spatial decimationmode in the structure shown in FIG. 28 is described below.

In the structure shown in FIG. 28, a memory 153 is additionally disposedbetween the motion detecting unit 121 and the decimation modedetermination unit 151 in the structure shown in FIG. 18. Use of thememory 153 makes it possible for the decimation mode determination unit151 to determine the decimation mode based on not only the moving speedv of a current frame but also the moving speed p of a previous frame.The manner in which the decimation mode determination unit 151determines the decimation mode is described below with reference to FIG.29.

First, the decimation mode determination unit 151 examines the movingspeed p of a block of an immediately previous frame received from thememory 153.

In a case in which the moving speed p is in the range 2≦p<c, if themoving speed v of a current frame received from the motion detectingunit 121 is within the range 2−Δv1≦v<−;c+Δv2, then the decimation modedetermination unit 151 selects a process 3 (a mode in which thedecimation position is shifted in a direction opposite to a direction inwhich a subject is moving) as shown in FIG. 29. When the moving speed pis not within the range described above, the process mode to be used isdetermined according to the criterion shown in FIG. 27.

In a case in which the moving speed p is in the range c≦p<d, if themoving speed v of the current frame received from the motion detectingunit 121 is within the range c−Δv3≧v<d+Δv4, then the decimation modedetermination unit 151 selects a process 1 (a mode in which thedecimation position is not shifted) as shown in FIG. 29. When the movingspeed p is not within the range described above, the process mode to beused is determined according to the criterion shown in FIG. 27.

In a case in which the moving speed p is in the range d≦p<f, if themoving speed v of the current frame received from the motion detectingunit 121 is within the range d−Δv5≦v<f+Δv6, then the decimation modedetermination unit 151 selects a process 2 (a mode in which thedecimation position is shifted in the same direction as the direction inwhich the subject is moving) as shown in FIG. 29. When the moving speedp is not within the range described above, the process mode to be usedis determined according to the criterion shown in FIG. 27.

In a case in which the moving speed p is in the range f≦p<e, if themoving speed v of the current frame received from the motion detectingunit 121 is within the range f−Δv7≦v<e+Δv8, then the decimation modedetermination unit 151 selects the process 3 (the mode in which thedecimation position is shifted in a direction opposite to the directionin which the subject is moving) as shown in FIG. 29. When the movingspeed p is not within the range described above, the process mode to beused is determined according to the criterion shown in FIG. 27.

In a case in which the moving speed p is in the range e≦p, if the movingspeed v of the current frame received from the motion detecting unit 121is within the range e−Δv9≦v, then the decimation mode determination unit151 selects the process 1 (the mode in which the decimation position isnot shifted) as shown in FIG. 29. When the moving speed p is not withinthe range described above, the process mode to be used is determinedaccording to the criterion shown in FIG. 27.

As described above, when the decimation mode determination unit 151determines the decimation method, if the moving speed of the currentframe is within a range wider by margins than a strict rangecorresponding to a decimation mode applied to the previous frame, thedecimation mode determination unit 151 determines that the current frameshould be decimated in the same decimation mode as that applied to theprevious frame. This prevents the decimation method from being switchedvery frequently with passage of time.

The margins Δv1 to Δv9 of the limits of the ranges of the moving speedmay be determined in many ways. For example, as shown in FIG. 30, a newthreshold value T1 lower than the threshold value T may be introduced,and margins may be defined by differences between new ranges determinedby the threshold value T1 and the corresponding ranges determined by thethreshold value T. For example, a margin Δv3 shown in FIG. 30 is givenby the absolute value of the difference between the lower limit c of arange determined by the threshold value T and the lower limit c′ of acorresponding range determined by the threshold value T1. A margin Δv4for the upper limit of the same range is given by the absolute value ofthe difference between the upper limit d of the range determined by thethreshold value T and the upper limit d′ of the corresponding rangedetermined by the threshold value T1.

Referring to FIG. 28, the structure of the block processing unit 131 andthe process performed by the block processing unit 131 are describedbelow. The memory 153 stores the result of the decimation modedetermined by the decimation mode determination unit 151 for a previousframe. The decimation mode determination unit 151 determines thedecimation mode in accordance with data indicating the correspondencebetween the moving speed of a subject and the image quality of datagenerated as a result of the decimation process performed in the mode inwhich the sample point position is not shifted, and in accordance withthe decimation mode used in a previous frame.

More specifically, in a case in which the decimation process for aprevious frame was performed in the fixed SPP mode and the moving speedof a subject detected by the motion detecting unit 121 corresponds to animage quality score equal to or higher than a predetermined thresholdvalue T1, the decimation mode determination unit 151 determines that thedecimation process should be performed in the fixed SPP mode. In a casein which the decimation process for a previous frame was performed inthe fixed SPP mode and the moving speed of a subject detected by themotion detecting unit 121 corresponds to an image quality score lessthan the predetermined threshold value T1, the decimation modedetermination unit 151 determines that the decimation process should beperformed in the SPP shifting mode. In a case in which the decimationprocess for a previous frame was performed in the SPP shifting mode andthe moving speed of a subject detected by the motion detecting unit 121corresponds to an image quality score less than a predeterminedthreshold value T2, the decimation mode determination unit 151determines that the decimation process should be performed in the SPPshifting mode. In a case in which the decimation process for a previousframe was performed in the SPP shifting mode and the moving speed of asubject detected by the motion detecting unit 121 corresponds to animage quality score equal to or higher than the predetermined thresholdvalue T2, the decimation mode determination unit 151 determines that thedecimation process should be performed in the fixed SPP mode.

The memory 153 also stores data indicating the mode in which thesampling point position was shifted or fixed in the spatial decimationperformed on the previous frame by the block processing unit 131. Theblock processing unit 131 performs spatial decimation such that when thevirtual moving speed of a subject obtained as a result of spatialdecimation performed on the previous frame depending on the mode inwhich the sampling point position was shifted or fixed is evaluatedbased on data indicating the correspondence between the moving speed ofa subject and the image quality score for data obtained as a result ofdecimation in the fixed SPP mode, if an image quality scorecorresponding to the virtual moving speed of the subject is equal to orhigher than a predetermined threshold value T3, then the spatialdecimation is performed in the same mode as the mode used in the spatialdecimation performed on the previous frame, but when the virtual movingspeed of the subject obtained as the result of spatial decimationperformed on the previous frame depending on the mode in which thesampling point position was shifted or fixed is evaluated based on dataindicating the correspondence between the moving speed of a subject andthe image quality score for data obtained as a result of decimation inthe fixed SPP mode, if the image quality score corresponding to thevirtual moving speed of the subject is lower than the predeterminedthreshold value T3, then the spatial decimation is performed in a modeselected such that the virtual speed of the subject obtained as a resultof the decimation falls within a range of the moving speed in which theimage quality score evaluated based on the data indicating thecorrespondence between the moving speed of a subject and the imagequality score for data obtained as a result of decimation in the fixedSPP mode is equal to or higher than a predetermined threshold value T4.

Before the operation of the block processing unit 132 is described, theoperation of the block processing unit 133 is described below.

The block processing unit 133 performs frame decimation (temporaldecimation) when the amount of motion of N blocks located at the sameposition of respective N successive frames supplied from the blockdistributor 122 of the motion detector 120 is less than 1 pixel in bothhorizontal and vertical directions.

More specifically, as shown in FIG. 31, the block processing unit 133performs decimation such that four blocks Bi at the same position ofrespective four successive frames F1 to F4 are decimated into one block(block Bi of frame F1 in the example shown in FIG. 9) identical to oneof these four blocks.

The resultant data of the four blocks whose total data size was reducedto ¼ of the original total data size via the temporal decimation issupplied from the block processing unit 133 to the output unit 140 (thatis, the data of one block is supplied to the output unit 140). Althoughin the present example N=4, the value of N is not limited to 4 but N mayhave another value.

Furthermore, although in the present example, the block Bi of the frameF1 is selected from blocks Bi of four frames F1 to F4 and the selectedblock Bi of the frame F1 is output as the surviving block, a block ofanother frame may be selected. Yet alternatively, a block may becalculated from four frames F1 and F4, and the resultant block may beoutput.

Now, the operation of the block processing unit 132 is described below.The block processing unit 132 performs pixel decimation (spatialdecimation) and frame decimation (temporal decimation) when the amountof motion of N blocks located at the same position of respective Nsuccessive frames supplied from the block distributor 122 of the motiondetector 120 is equal to or greater than 1 pixel but less than 2 pixelsin both horizontal and vertical directions.

The moving speed (v=1 to 2) per frame of blocks supplied to the blockprocessing unit 132 satisfies equations (1) and (2), that is, the movingspeed satisfies the condition necessary to obtain the super resolutioneffect. However, if decimation is performed in the fixed SPP mode in asimilar manner as described earlier with reference to FIGS. 1 and 12,there is a possibility that perceptible degradation in image qualityoccurs.

To avoid the above problem, the block processing unit 132 performsspatial decimation by a factor of 2 (m=2) rather than a factor of 4(m=4). In the decimation process, as with the block processing unit 131,the block processing unit 132 selects the fixed SPP mode or the SPPshifting mode depending on the moving speed of a subject.

The decimation process by a factor of 2 (m=2) in the fixed SPP mode andthat in the SPP shifting mode are described in detail below withreference to FIGS. 32 to 37.

FIGS. 32 to 34 show examples of decimation processes performed by theblock processing unit 132 when the moving speed v in the horizontaldirection is 1 to 2 pixels/frame.

In the example shown in FIG. 32, decimation is performed in the fixedSPP mode such that pixel values p1 to p4 of respective frames (k-th to(k+3)-th frames) are decimated into two pixel values equal to two pixelvalues (p1 and p3 in this specific example) selected from pixel valuesp1 to p4. In other words, two pixels are decimated into one pixel (thatis, the decimation factor m=2).

In the example shown in FIG. 33, decimation is performed in the SPPshifting mode such that pixel values p1 to p4 of first two frames (k-thand (k+1)-th frames) are decimated into pixel values p1 and p3, andpixel values p1 to p4 of the following two frames ((k+2)-th and (k+3)-thframes) are decimated into pixel values p2 and p4. In the example shownin FIG. 34, decimation is also performed in the SPP shifting mode, butthe decimation is performed such that pixel values p1 to p4 of first twoframes (k-th and (k+1)-th frames) are decimated into pixel values p2 andp4, and pixel values p1 to p4 of the following two frames ((k+2)-th and(k+3)-th frames) are decimated into pixel values p1 and p3.

FIGS. 35 to 37 show examples of decimation processes performed by theblock processing unit 132 when the moving speed v in the verticaldirection is 1 to 2 pixels/frame.

In the example shown in FIG. 35, decimation is performed in the fixedSPP mode such that pixel values p1 to p4 of respective frames (k-th to(k+3)-th frames) are decimated into two pixel values equal to two pixelvalues (p1 and p3 in this specific example) selected from pixel valuesp1 to p4. In other words, two pixels are decimated into one pixel (thatis, the decimation factor m=2).

In the example shown in FIG. 36, decimation is performed in the SPPshifting mode such that pixel values p1 to p4 of first two frames (k-thand (k+1)-th frames) are decimated into pixel values p1 and p3, andpixel values p1 to p4 of the following two frames ((k+2)-th and (k+3)-thframes) are decimated into pixel values p2 and p4. In the example shownin FIG. 37, decimation is also performed in the SPP shifting mode, butthe decimation is performed such that pixel values p1 to p4 of first twoframes (k-th and (k+1)-th frames) are decimated into pixel values p2 andp4, and pixel values p1 to p4 of the following two frames ((k+2)-th and(k+3)-th frames) are decimated into pixel values p1 and p3.

As with the block processing unit 131, the block processing unit 132 hasa structure similar to that shown in FIG. 18. That is, the blockprocessing unit 132 includes a decimation mode determination unit 151and a decimation execution unit 152. The data indicating the amount ofmotion output from the motion detecting unit 121 is applied to thedecimation mode determination unit 151. As described earlier, blockssupplied to the block processing unit 132 are those blocks whose movingspeed v is 1 to 2 pixels/frame. Thus, the moving speed v indicated bythe data input to the decimation mode determination unit 151 of theblock processing unit 132 is 1 to 2 pixels/frame.

Based on the moving speed, the decimation mode determination unit 151selects one of modes shown in FIGS. 32 to 37 in which to performdecimation. In accordance with the determination made by the decimationmode determination unit 151, the decimation execution unit 152 executesthe decimation process in the mode selected from those shown in FIGS. 32to 37.

The details of the decimation mode determination process performed bythe decimation mode determination unit 151 of the block processing unit132 are described below with reference to FIGS. 38 to 40. FIGS. 38 to 40are graphs, similar to those shown in FIGS. 22 to 24, showing thequality of image data reproduced from data produced by performingdecimation (spatial/temporal decimation) in the fixed SPP mode as shownin FIG. 32 or 35. In these graphs, the image quality based on subjectiveevaluation is plotted as a function of the moving speed of a subject.The horizontal axis represents the moving speed of the subject, and thevertical axis represents the image quality score.

In FIG. 38, as in FIG. 20, T denotes a threshold value of the imagequality score. When the score is higher than the threshold value T, theimage quality is regarded as good. In the example shown in FIG. 38,image quality scores higher than the threshold value T are obtained Inregions denoted by A. In regions B other than the regions A, the imagequality score is lower than the threshold value T. When a subject has amoving speed in one of these regions B, if decimation is performed inthe fixed SPP mode, the super resolution effect cannot be obtained atall or the super resolution effect is insufficient even if it isobtained.

When the moving speed is in one of regions B, decimation is performed inthe SPP shifting mode such that the moving speed of the subject isvirtually accelerated or decelerated. More specifically, when thesubject is moving in a horizontal direction, decimation is performed inthe SPP shifting mode as shown in FIG. 33 or 34. When the subject ismoving in a vertical direction, decimation is performed in the SPPshifting mode as shown in FIG. 36 or 37. By performing decimation insuch a manner, as high image quality as that obtained in the regions Acan be obtained.

In addition to the spatial decimation, the block processing unit 132also performs temporal decimation. The temporal decimation is performedin a similar manner as described earlier with reference to FIG. 12. Morespecifically, for example, two frames are selected from successive fourframes, and blocks of the four frames are decimated into blocks of theselected two frames. In a case in which temporal decimation is performedbefore spatial decimation is performed, data applied to the spatialdecimation process has a frame rate one-half the original frame rate,and thus the effective moving speed of a subject in the decimationprocess in the SPP shifting mode becomes ±0.5 pixels/frame.

Regions B-1 and B-2 shown in FIG. 39 are part of the regions B shown inFIG. 38. In these regions B-1 and B-2, the image quality score is lessthan the threshold value, because the moving speed v of a subject doesnot have a value at which a sufficient super resolution effect occurs.When a subject moves at a speed in the range B-1 or B-2, if the subjectis virtually accelerated (by +Δv1 or +Δv2), that is, if decimation isperformed while shifting the decimation position in a direction oppositeto a direction in which the subject is actually moves the region B-1shown in FIG. 39 is effectively moved to a region C-1 and the region B-2is effectively moved to a region C-2 As a result, the virtual movingspeed per frame, v, of the subject is moved into one of regions (C-1 andC-1 shown in FIG. 39) in which the image quality score can be equal toor greater than the threshold value T. That is, by performing decimationwhile shifting the sampling point position to the left as the frameadvances as shown in FIG. 33, 34, 36, or 37, it is possible to performimage data conversion (compression) without significant degradation inimage quality.

Similarly, when a subject is moving at a speed in a range B-3 shown inFIG. 40, if the subject is virtually decelerated (by −Δv3), that is, ifdecimation is performed while shifting the decimation position in thesame direction as a direction in which the subject is actually moves(for example, when the subject is moving to the right, the decimationprocess is performed as shown in FIG. 33), the region B-3 shown in FIG.40 is effectively moved to a region C-3. As a result, the virtual movingspeed per frame, v, of the subject falls within the region (C-3 shown inFIG. 40) in which the image quality score can be equal to or greaterthan the threshold value T, and thus image data conversion (compression)is performed without causing significant degradation in image quality.

In the motion image data conversion apparatus 100 according to thepresent embodiment, the block processing unit 132 performs spatial pixeldecimation by a factor of 2 only for blocks whose motion in thehorizontal or vertical direction is equal to or greater than 1pixel/frame but less than 2 pixels/frame, and thus blocks whose movingspeed v is less than 1 pixels/frame in the graphs shown in FIGS. 38 to40 are not subjected to the process by the block processing unit 132.However, as described earlier, the above-described criterion used by theblock dividing unit 122 to determine which one of the block processingunits 131 to 133 of the block processor 130 to supply the block to ismerely one example, and the block distributor 122 may supply the blockwhose motion is less than 1 pixel/frame to the block processing unit 132to perform spatial decimation in one of decimation modes described abovewith reference to FIGS. 32 to 37. Therefore, the moving speed v lessthan 1 pixel/frame is also included in the graphs shown in FIGS. 32 to37.

Note that the image quality evaluation curves shown in FIGS. 38 to 40are merely examples based on a subjective evaluation result, and theimage quality may be evaluated in many ways other than that describedabove and the decimation mode may be determined in accordance with theevaluation result.

In the block processing unit 132 shown in FIG. 14 responsible forprocessing block data with motion equal to or greater than 1 pixel/framebut less than 2 pixels/frame, the decimation mode determination unit 151shown in FIG. 18 determines which one of modes described below should beused based on the amount of motion indicated by motion data receivedfrom the motion detector, and the decimation execution unit 152 shown inFIG. 18 executes decimation in the determined mode.

(a) Decimation is performed while fixing sampling point positions (FIGS.32 and 35).

(b) Decimation is performed while shifting the sampling point positionin the same direction as a direction in which a subject is moving, asthe frame advances (FIGS. 33, 34, 36, and 37).

(c) Decimation is performed while shifting the sampling point positionin a direction opposite to a direction in which a subject is moving, asthe frame advances (FIGS. 33, 34, 36, and 37).

When the decimation mode determination unit 151 shown in FIG. 18determines the decimation mode, for example, the image qualityevaluation curves shown in FIGS. 38 to 40 are used. That is, when themoving speed is in a region in which an image quality score equal to orgreater than the threshold value T is obtained judging from the imagequality curve, decimation is performed in the fixed SPP mode, while whenthe moving speed is in a region in which the image quality scoreestimated from the image quality curve is less than the threshold valueT, decimation is performed in the SPP shifting mode such that the amountof virtual motion falls within a range in which the image qualityestimated based on the image quality curve is equal to or greater thanthe threshold value T.

The block processing unit 132 selects a proper decimation mode dependingon the moving speed v per frame in the horizontal direction indicated bymotion data received from the motion detecting unit 121. FIG. 41A is atable indicating the correspondence between the moving speed v and thedecimation mode. FIG. 41B is a table representing the details of therespective decimation modes described in the table shown in FIG. 41A. Inthe table shown in FIG. 41A, values of i, j, m, and k corresponds tovalues of the moving speed indicated on the horizontal axes of FIGS. 38to 40.

The decimation processing modes shown in FIG. 41A are described below.

When 1≦v<i, decimation is performed in the mode 3 in which decimationposition is shifted in a direction opposite to a direction in which asubject is moving.

When i≦v<j, decimation is performed in the mode 1 in which decimationposition is not shifted. When j≦v<m, decimation is performed in the mode2 in which decimation position is shifted in the same direction as adirection in which a subject is moving.

When m≦v<k, decimation is performed in the mode 3 in which decimationposition is shifted in a direction opposite to a direction in which asubject is moving.

When k≦v, decimation is performed in the mode 1 in which decimationposition is not shifted.

By changing the decimation mode depending on the moving speed v perframe of a subject included in a block in the above-described manner, itis possible to virtually change the moving speed of the subject suchthat the virtual moving speed falls within the range A shown in FIG. 38,thereby achieving image data conversion in a manner in which imagequality score equal to or higher than the threshold value T isguaranteed.

The block processing unit 132 performs decimation in one of modesdepending on the moving speed v per frame of a subject included in ablock as described above, and outputs resultant data to the output unit140 (FIGS. 13 and 14). In addition to the spatial decimation by a factorof 2, the block processing unit 132 also performs temporal decimation(to decimate four frames into 2 frames) as shown in FIG. 12, and thusthe data size of each block is reduced to ¼ of the original data size,and the total data size of four blocks is also reduced to ¼ of theoriginal total data size. The resultant data of 4 blocks with the datasize reduced by the factor of 4 is supplied from the block processingunit 131 to the output unit 14.

Either one of the spatial decimation and the temporal decimation may beperformed first, and the other may be performed next, because the sameresult can be obtained regardless of the order of processing. Theprocess described above can be performed for N/2 frames and N/2 adjacentpixels when N is a positive even number, and thus it is not necessarilyneeded to perform the process in units of 2 frames or in units of 2pixels.

Although in the above-described example of the decimation processperformed by the block processing unit 131, the decimation factor m=4and the number of frames N=4, the parameters are not limited to thesevalues. Note that the image quality evaluation curves 215 indicating theimage quality as a function of the moving speed of a subject shown inFIGS. 38 to 40 are for the decimation factor m=4 and the number offrames N=4, and the image quality evaluation curve varies depending onthe values of the parameters. Thus, it is necessary to determine animage quality evaluation curve for the specific parameter valuesemployed, and determine the decimation mode depending on the determinedimage quality evaluation curve.

Now, the output unit 140 is described below. If the output unit 140receives data of blocks with reduced data sizes from the blockprocessing units 131 to 133 of the block processor 130, the output unit140 outputs the received data together with data indicating the mannerin which each block was processed. The data indicating the manner ofprocessing includes data indicating which one of the spatial decimationmode, the temporal decimation mode, and the spatial/temporal decimationmode the processing was performed in, data indicating whether shiftingof the decimation position was performed when the spatial decimation wasperformed, data indicating a manner in which the decimation position wasshifted when the decimation position was shifted, and data indicatingthe frame rate and the spatial resolution of an original motion image.Note that other data may be included in the data indicating the mannerof processing.

Now, referring to a flow chart shown in FIG. 42, the operation of themotion detector 120 and the block processor 130 of the motion image dataconversion apparatus 100 is described below.

In step S1, blocks of a P-th frame of four frames successively input tothe motion image data conversion apparatus 100 are supplied to themotion detecting unit 121 of the motion detector 120 from the blockdividing unit 112 of the block divider 110. Furthermore, the motiondetecting unit 121 of the motion detector 120 receives an M-th frame(reference frame) from the image storage unit 111.

In step S2, the motion detecting unit 121 selects one of the blocksincluded in the P-th frame as a block of interest and detects the motionvector of the block of interest with respect to the reference frame. Themotion detector 121 supplies the detected motion vector to the blockdistributor 122.

In step S3, the block distributor 122 determines whether the amount ofmotion, in the horizontal (X) direction or the vertical (Y) direction inwhich the amount of motion is greater than in the other direction,indicated by the motion vector supplied from the motion detecting unit121 is equal to or greater than 2 pixels, the block distributor 122. Ifit is determined that at least the amount of motion in one of thehorizontal (X) and vertical (Y) directions is equal to or greater than 2pixels, the process proceeds to step S4.

In step S4, a determination as to the decimation mode is made by thedecimation mode determination unit 151 (shown in FIG. 18) of the blockprocessing unit 131 of the block processor 130. More specifically, basedon the amount of motion, the decimation mode determination unit 151determines which one of the modes shown in FIGS. 15 to 17 (when asubject is moving in a horizontal direction) or modes shown in FIGS. 24to 16 (when the subject is moving in a vertical direction) thedecimation should be performed in. That is, depending on the amount ofmotion indicated by the data supplied from the motion detecting unit121, the decimation mode determination unit 151 determines whether toperform decimation in the fixed SPP mode (FIGS. 15 and 24) or the SPPshifting mode (FIGS. 16, 17, 25, and 26).

When the decimation mode determination unit 151 determines thedecimation mode, for example, the image quality evaluation curves shownin FIGS. 20 to 22 are used. More specifically, when the amount of motion(per frame) in the horizontal (X) or vertical (Y) direction indicated bythe motion vector supplied from the motion detecting unit 121 is withinone of ranges in which the image quality score is equal to or greaterthan the threshold value T, the decimation is performed in the fixed SPPmode (FIG. 15 or 24). On the other hand, when the amount of motion (perframe) in the horizontal (X) or vertical (Y) direction indicated by themotion vector supplied from the motion detecting unit 121 is within oneof ranges in which the image quality score is less than the thresholdvalue T, the decimation is performed in the SPP shifting mode (FIG. 16,17, 25, or 26) such that the amount of virtual motion falls within arange in which the image quality estimated based on the image qualitycurve is equal to or greater than the threshold value T.

If it is determined in step S4 that the amount of motion (per frame) inthe horizontal (X) or vertical (Y) direction indicated by the motionvector supplied from the motion detecting unit 121 is within one ofranges in which the image quality score is equal to or greater than thethreshold value T, then the decimation mode determination unit 151determines, judging from the amount of motion, that decimation should beperformed without shifting the decimation position. In this case, theprocess proceeds to step S6. In step S6, the decimation execution unit152 shown in FIG. 18 executes a decimation process in the fixed SPP mode(FIG. 15 or 24).

On the other hand, if it is determined that the amount of motion (perframe) in the horizontal (X) or vertical (Y) direction indicated by themotion vector supplied from the motion detecting unit 121 is not withinany one of ranges in which the image quality score is equal to orgreater than the threshold value T, the decimation mode determinationunit 151 determines that shifting of the decimation position should beperformed. In this case, the process proceeds to step S5, and thedecimation execution unit 152 shown in FIG. 18 executes a decimationprocess in the SPP shifting mode (FIG. 16, 17, 25, or 26). Thedecimation mode determination unit 151 also determines whether thesample position is shifted in the same direction as a direction in whicha subject is moving or in an opposite direction, and the decimationexecution unit 152 (FIG. 18) executes the decimation process inaccordance with the determination made by the decimation modedetermination unit 151.

That is, in step S5 or S6, the block processing unit 131 performsdecimation on the four blocks supplied from the block distributor 122 asshown in FIG. 15, 16, or 17 or in FIG. 24, 25, or 26 such that thenumber of pixels is reduced by a factor of 4. The resultant data of fourblocks whose data size was reduced to ¼ of the original data size issupplied to the output unit 140.

Note that in the example described above, it is assumed that thedecimation factor m is equal to 4 and the number N of frames processedat a time is equal to 4. The values of the parameters are not limited tothose employed above, but other values may be employed. When theparameters have other values, data is compressed by a different factordepending on the values of the parameters employed. Note that asdescribed earlier with reference to FIGS. 20 to 22, the image qualityevaluation curve indicating the image quality as a function of themoving speed of a subject varies depending on the values of parameters.Thus, it is necessary to determine an image quality evaluation curve forthe specific parameter values employed, and determine the decimationmode depending on the determined image quality evaluation curve.

In a case in which the block distributor 122 determines in step S3 thatthe amount of motion indicated by the motion vector supplied from themotion detecting unit 121 is not equal to or greater than 2 pixels/framein either the horizontal (X) direction or vertical (Y) direction, theprocess proceeds to step S7. In step S7, if the block distributor 122further determines that the amount of motion (per frame) indicated bythe motion vector supplied from the motion detecting unit 121 is lessthan 1 pixel/frame in both the horizontal (X) direction and the vertical(Y) direction, the process proceeds to step S8.

In step S8, only temporal decimation is performed by the blockprocessing unit 133 of the block processor 130. More specifically, theblock processing unit 133 performs frame decimation (temporaldecimation) on a total of N blocks of successive N frames (whose motionis less than 1 pixel/frame both in the horizontal direction and in thevertical direction) supplied from the block distributor 122 of themotion detector 120.

More specifically, as described earlier with reference to FIG. 31, theblock processing unit 133 performs frame decimation such that fourblocks Bi at the same position of respective four successive frames F1to F4 are decimated into one block (block Bi of frame F1 in the exampleshown in FIG. 9) selected from thee four blocks.

The resultant data of the four blocks whose total data size was reducedto ¼ of the original total data size via the temporal decimation issupplied from the block processing unit 133 to the output unit 140 (thatis, the data of one block is supplied to the output unit 140). Althoughin the present example N=4, the value of N is not limited to 4 but N mayhave another value.

Furthermore, although in the present example, the block Bi of the frameF1 is selected from blocks Bi of four frames F1 to F4 and the selectedblock Bi of the frame F1 is output as the surviving block, a block ofanother frame may be selected. Yet alternatively, a block may becalculated from four frames F1 and F4, and the resultant block may beoutput.

In a case in which the block distributor 122 determines in step S7 thatthe amount of motion in either the horizontal (X) or vertical (Y)direction indicated by the motion vector supplied from the motiondetecting unit 121 is less than 1 pixel/frame, the process proceeds tostep S9.

In step S9, a determination as to the decimation mode is made by thedecimation mode determination unit 151 (shown in FIG. 18) of the blockprocessing unit 132 of the block processor 130. More specifically, thedecimation mode determination unit 151 determines which one of the modesshown in FIGS. 32 to 34 (when a subject is moving in a horizontaldirection) or modes shown in FIGS. 35 to 37 (when the subject is movingin a vertical direction) the decimation should be performed in. That is,depending on the amount of motion indicated by the data supplied fromthe motion detecting unit 121, the decimation mode determination unit151 determines whether to perform decimation in the fixed SPP mode (FIG.32 or 35) or the SPP shifting mode (FIG. 33, 34, 36, or 37).

When the decimation mode determination unit 151 determines thedecimation mode, for example, the image quality evaluation curves shownin FIGS. 38 to 40 are used. More specifically, when the amount of motion(per frame) in the horizontal (X) or vertical (Y) direction indicated bythe motion vector supplied from the motion detecting unit 121 is withinone of ranges in which the image quality score is equal to or greaterthan the threshold value T, the decimation is performed in the fixed SPPmode (FIG. 32 or 35). On the other hand, when the amount of motion (perframe) in the horizontal (X) or vertical (Y) direction indicated by themotion vector supplied from the motion detecting unit 121 is within oneof ranges in which the image quality score is less than the thresholdvalue T, the decimation is performed in the SPP shifting mode (FIG. 33,34, 36, or 37) such that the amount of virtual motion falls within arange in which the image quality estimated based on the image qualitycurve is equal to or greater than the threshold value T.

If it is determined in step S9 that the amount of motion (per frame) inthe horizontal (X) or vertical (Y) direction indicated by the motionvector supplied from the motion detecting unit 121 is within one ofranges in which the image quality score is equal to or greater than thethreshold value T, the decimation mode determination unit 151 determinesthat shifting of the decimation position should not be performed. Inthis case, the process proceeds to step S11, and the decimationexecution unit 152 shown in FIG. 18 executes a spatial decimationprocess in the fixed SPP mode (FIG. 32 or 35). Furthermore, thedecimation execution unit 152 executes a temporal decimation process insuch a manner as described earlier with reference to FIG. 12. Note thateither one of the spatial decimation and the temporal decimation may beperformed first, and the other process may be performed next.

On the other hand, if it is determined that the amount of motion (perframe) in the horizontal (X) or vertical (Y) direction indicated by themotion vector supplied from the motion detecting unit 121 is not withinany one of ranges in which the image quality score is equal to orgreater than the threshold value T, the decimation mode determinationunit 151 determines that shifting of the decimation position should beperformed. In this case, the process proceeds to step S10, and thedecimation execution unit 152 shown in FIG. 18 executes a decimationprocess in the SPP shifting mode (FIG. 33, 34, 36, or 37). Furthermore,the decimation execution unit 152 executes a temporal decimation processin such a manner as described earlier with reference to FIG. 12. Notethat either one of the spatial decimation and the temporal decimationmay be performed first, and the other process may be performed next.

The decimation mode determination unit 151 also determines whether thesample position is shifted in the same direction as a direction in whicha subject is moving or in an opposite direction, and the decimationexecution unit 152 (FIG. 18) executes the decimation process inaccordance with the determination made by the decimation modedetermination unit 151.

That is, in step S10 or step S11, the block processing unit 132 performsthe spatial decimation by a factor of m=2 and also the temporaldecimation on the four blocks supplied from the block distributor 122,and the resultant data of four blocks whose data size was reduced to ¼of the original data size is supplied to the output unit 140.

Note that, as described above, the values of parameters such as thedecimation factor m and the number N of frames processed at a time arenot limited to those employed above, but other values may be employed.When the parameters have other values, data is compressed by a differentfactor depending on the values of the parameters employed. Note that asdescribed earlier with reference to FIGS. 38 to 40, the image qualityevaluation curve indicating the image quality as a function of themoving speed of a subject varies depending on the values of parameters.Thus, it is necessary to determine an image quality evaluation curve forthe specific parameter values employed, and determine the decimationmode depending on the determined image quality evaluation curve.

The above-described process is performed each time blocks of N (forexample, four) frames are supplied from the block divider 110.

At a stage following the output unit 140 of the motion image dataconversion apparatus 100 shown in FIG. 13, there is disposed a storagemedium such as a hard disk or a DVD or a network output device so thatthe compressed data obtained via the above-described process is storedor output over a network.

(3) Apparatus and Method of Reproducing Motion Image Data

A motion image reproduction apparatus for reproducing a motion imagefrom compressed data produced by the motion image data conversionapparatus 100 and a method of reproducing a motion image are describedbelow.

FIG. 43 shows the structure of the motion image reproduction apparatus300. As shown in FIG. 43, the motion image reproduction apparatus 300includes a distributor 310, block decompressors 321 to 323, and acombiner 330.

Compressed data produced by the motion image data conversion apparatus100 described above and associated attributed data necessary toreproduce the image from the compressed data are input to thedistributor 310. The attribute data includes not only attribute data ofblocks but also data indicating the manner in which blocks wereprocessed, that is, data indicating whether spatial decimation ortemporal decimation was performed, whether the spatial decimation wasperformed in the fixed SPP mode or the SPP shifting mode if the spatialdecimation was performed, and in which direction the sample pointposition was shifted if the spatial decimation was performed in the SPPshifting mode.

In accordance with the manner, indicated by the input attribute data, inwhich blocks were processed, the distributor 310 supplies the compresseddata to be decompressed and the attribute data indicating the manner inwhich the blocks were process to one of the block decompressors 321 to323.

More specifically, in a case in which the blocks are those processed bythe block processing unit 131 of the motion image data conversionapparatus 100 shown in FIGS. 13 and 14, the distributor 310 supplies, tothe block decompressor 321, the block data subjected to the spatialdecimation together with data indicating which direction the spatialdecimation was performed in, whether the decimation position wasshifted, and whether which direction the decimation position was shiftedin.

On the other hand, in a case in which the blocks are those processed bythe block processing unit 132 of the motion image data conversionapparatus 100, the distributor 310 supplies similar data to the blockdecompressor 322. In a case in which the blocks are those processed bythe block processing unit 133 of the motion image data conversionapparatus 100, the distributor 310 supplies similar data to the blockdecompressor 323. Note that there is no particular restriction on thedata associated with blocks and the data indicating the manner in whichblocks were processed, as long as data indicates information sufficientto reproduce a motion image.

The block decompressor 321 decompresses the data subjected to thespatial decimation performed by the block processing unit 131 of themotion image data conversion apparatus 100. The decompression isperformed as shown in FIG. 44A or 44B in accordance with the datasupplied from the distributor 310 and indicating which direction thespatial decimation was performed in, whether shifting of the decimationposition was performed, and which direction the decimation position wasshifted in, thereby reconstructing the block. The resultantreconstructed block and data indicating the mode in which the processwas performed are output to the combiner 330.

The process shown in FIGS. 44A and 44B are described in further detailbelow. FIG. 44A shows the process performed by the block decompressor321 of the motion image reproduction apparatus 300 to decompress datasubjected to the spatial decimation in the horizontal direction by theblock processing unit 131 of the motion image data conversion apparatus100. For example, when the original block has a size of 8×8 pixels, thedata supplied to the block decompressor 321 has a data size one-quarterthe original data size, that is, the data supplied to the blockdecompressor 321 includes 16 pixels. The block decompressor 321 expandseach pixel into 1×4 pixels by performing a process (a-1) shown in FIG.44. Although in the process (a-1) shown in FIG. 44, the given pixelvalue is directly used as pixel values of all four expanded pixels,pixel values of four expanded pixels may be calculated from the givenpixel value and other pixel values.

In a case in which decimation in the horizontal direction was performed,the block decompressor 321 further performs a process (a-2) shown inFIG. 44. More specifically, a set of 1×4 pixels obtained via the process(a-1) is laid as shown in (a-2) thereby reproducing a block including8×8 pixels from the input 16 pixels. The resultant reproduced data isoutput to the combiner 330 together with data indicating the manner inwhich decimation was performed.

FIG. 44B shows a process performed by the block decompressor 321 todecompress data subjected to the spatial decimation in the verticaldirection by the block processing unit 131 of the motion image dataconversion apparatus 100. For example, when the original block has asize of 8×8 pixels, the data supplied to the block decompressor 321 hasa data size one-quarter the original data size, that is, the datasupplied to the block decompressor 321 includes 16 pixels. The blockdecompressor 321 expands each pixel into 4×1 pixels by performing aprocess (b-1) shown in FIG. 44. Although in the process (b-1) shown inFIG. 44, the given pixel value is directly used as pixel values of allfour expanded pixels, pixel values of four expanded pixels may becalculated from the given pixel value and other pixel values.

In a case in which decimation in the vertical direction was performed,the block decompressor 321 further performs a process (b-2) shown inFIG. 44. More specifically, a set of 4×1 pixels obtained via the process(b-1) shown in FIG. 44 is laid as shown in (b-2) thereby reproducing ablock including 8×8 pixels from the input 16 pixels. The resultantreproduced data is output to the combiner 330 together with dataindicating the manner in which decimation was performed.

Now, a process performed by the block decompressor 323 is described. Theblock decompressor 323 decompresses the data subjected to the temporaldecimation performed by the block processing unit 133 of the motionimage data conversion apparatus 100.

As shown in FIG. 45, the block decompressor 323 produces block data of aplurality of frames from block data of one frame. More specifically, theblock decompressor 323 has a counter whose initial value is set to 0.Each time reproduction of one frame of data is completed, the counter isincremented by 1. When the counter value reaches 4 (=N (when m=4)), thecounter value is reset to 0.

When block data to be expanded is received from the data distributor310, the block data is stored in the memory of the data decompressor323, and block data of a plurality of frames is produced by copying theblock data stored in the memory until the counter value reaches theallowable upper limit preset depending on the decimation factor. Theresultant data is output to the mixer 330. In the example shown in FIG.45, the decimation factor m=4, and thus block data for four frames isproduced from one block data, and the produced data is output to themixer 330.

Now, a process performed by the block decompressor 322 is describedbelow. The block decompressor 323 is responsible for decompression ofdata subjected to the spatial and temporal decimation performed by theblock processing unit 132 of the motion image data conversion apparatus100. Note that either one of the spatial decompression and the temporaldecompression may be performed first, and the other process may beperformed next.

FIG. 46 shows an example of a manner in which the block decompressor 322performs temporal decompression. The block decompressor 322 has acounter whose initial value is set to 0. Each time reproduction of oneframe of data is completed, the counter is incremented by 1. When thecounter value reaches 2 (=N/2 (when temporal decimation factor m=2)),the counter value is reset to 0.

When block data to be decompressed is received from the data distributor310, the block data is stored in the memory of the data decompressor322, and block data of a plurality of frames is produced by copying theblock data stored in the memory until the counter value reaches theallowable upper limit preset depending on the decimation factor. Theresultant data is output to the mixer 330. In the example shown in FIG.46, the temporal decimation factor m=2, and thus block data for twoframes is produced from one block data.

The block decompressor 322 performs a spatial decompression process asfollows. Data expansion (decompression) is performed as shown in FIG.47A or 47B in accordance with the data (supplied from the distributor310) indicating which direction the spatial decimation was performed in,whether shifting of the decimation position was performed, and whichdirection the decimation position was shifted in, thereby reconstructingthe block.

The process shown in FIGS. 47A and 47B are described in further detailbelow. FIG. 47A shows the process performed by the block decompressor322 of the motion image reproduction apparatus 300 to decompress datasubjected to the spatial decimation in the horizontal direction by theblock processing unit 132 of the motion image data conversion apparatus100. For example, when the original block has a size of 8×8 pixels, thedata supplied to the block decompressor 321 has a data size one-half theoriginal data size, that is, the data supplied to the block decompressor321 includes 32 pixels. The block decompressor 321 expands each twopixels into 1×4 pixels by performing a process (a-1) shown in FIG. 47.Although in the process (a-1) shown in FIG. 47, the given pixel valuesof two pixels are directly used to produce four expanded pixels, pixelvalues of four expanded pixels may be calculated from the given pixelvalue and other pixel values.

In a case in which decimation in the horizontal direction was performed,the block decompressor 322 further performs a process (a-2) shown inFIG. 47. More specifically, a set of 1×4 pixels obtained via the process(a-1) is laid as shown in (a-2) thereby reproducing a block including8×8 pixels from the input 32 pixels. The resultant reproduced data isoutput to the combiner 330 together with data indicating the manner inwhich decimation was performed.

FIG. 47B shows a process performed by the block decompressor 322 todecompress data subjected to the spatial decimation in the verticaldirection by the block processing unit 132 of the motion image dataconversion apparatus 100. For example, when the original block has asize of 8×8 pixels, the data supplied to the block decompressor 321 hasa data size one-half the original data size, that is, the data suppliedto the block decompressor 321 includes 32 pixels. The block decompressor321 expands each two pixels into 4×1 pixels by performing a process(b-1) shown in FIG. 47. Although in the process (b-1) shown in FIG. 47,the given pixel values of two pixels are directly used to produce fourexpanded pixels, pixel values of four expanded pixels may be calculatedfrom the given pixel value and other pixel values.

In a case in which decimation in the vertical direction was performed,the block decompressor 322 further performs a process (b-2) shown inFIG. 47. More specifically, a set of 4×1 pixels obtained via the process(b-1) shown in FIG. 47 is laid as shown in (b-2) thereby reproducing ablock including 8×8 pixels from the input 32 pixels. The resultantreproduced data is output to the combiner 330 together with dataindicating the manner in which decimation was performed.

When the number of blocks input to the combiner 330 from the blockdecompressors 321 to 323 has reached a value that allows one completeframe to be reproduced, the combiner 330 lays the received blocks in aproper manner in accordance with the data which was received togetherwith the blocks and which indicates the manner in which the decimationwas performed, thereby reproducing one complete frame. The resultantreproduced one frame of data is output from the combiner 330. The blocklaying process is described in further detail below with reference toFIGS. 48 to 51.

FIGS. 48 to 51 show examples of data of one frame including a pluralityof blocks. In these examples shown in FIGS. 48 to 51, one frame of datais composed of 3×3 blocks, that is, 9 blocks. In these figures, eachsquare bordered by dotted lines is a block. A shaded square (forexample, a block 401 in FIG. 48) denotes a block of interest that wasreceived from one of the block decompressors 321 to 323 and is to belaid by the combiner 330.

FIG. 48 shows a manner of laying a block that was supplied to thecombiner 330 from one of the block decompressor 321 to 323 and that isto be laid by the combiner 330 is a block that was not subjected toshifting of the decimation position. In this case, the combiner 330 laysthe block at the same position as the original position at which theblock was divided by the block divider 110 of the motion image dataconversion apparatus 100 shown in FIGS. 13 and 14.

FIGS. 49A to 49D show examples of manners of laying blocks supplied fromone of the block decompressor 321 to 323 in a case in which the blockswere subjected to the spatial decimation in which the sampling pointposition was shifted horizontally.

The block shown in FIG. 49A is a block of a first frame (when motionimage data is processed in units of 4 (=N) frames) reproduced from blockdata that was subjected to the decimation in which the sampling pointposition was shifted to the right with frame advance, in the manner asdescribed earlier with reference to FIG. 16, or the block shown in FIG.49A is a block of a fourth frame (when motion image data is processed inunits of 4 (=N) frames) reproduced from block data that was subjected tothe decimation in which the sampling point position was shifted to theleft with frame advance, in the manner as described earlier withreference to FIG. 17. In this case, the block is laid at a positionshifted to the left by one pixel from the position of the input data.

The block shown in FIG. 49B is a block of a second frame (when motionimage data is processed in units of 4 (=N) frames) reproduced from blockdata that was subjected to the decimation in which the sampling pointposition was shifted to the right with frame advance, in the manner asdescribed earlier with reference to FIG. 16, or the block shown in FIG.49B is a block of a third frame (when motion image data is processed inunits of 4 (=N) frames) reproduced from block data that was subjected tothe decimation in which the sampling point position was shifted to theleft with frame advance, in the manner as described earlier withreference to FIG. 17. In this case, the block is laid at the sameposition as the position of the input data.

The block shown in FIG. 49C is a block of a third frame (when motionimage data is processed in units of 4 (=N) frames) reproduced from blockdata that was subjected to the decimation in which the sampling pointposition was shifted to the right with frame advance, in the manner asdescribed earlier with reference to FIG. 16, or the block shown in FIG.49C is a block of a second frame (when motion image data is processed inunits of 4 (=N) frames) reproduced from block data that was subjected tothe decimation in which the sampling point position was shifted to theleft with frame advance, in the manner as described earlier withreference to FIG. 17. In this case, the block is laid at a positionshifted to the right by one pixel from the position of the input data.

The block shown in FIG. 49D is a block of a fourth frame (when motionimage data is processed in units of 4 (=N) frames) reproduced from blockdata that was subjected to the decimation in which the sampling pointposition was shifted to the right with frame advance, in the manner asdescribed earlier with reference to FIG. 16, or the block shown in FIG.49D is a block of a first frame (when motion image data is processed inunits of 4 (=N) frames) reproduced from block data that was subjected tothe decimation in which the sampling point position was shifted to theleft with frame advance, in the manner as described earlier withreference to FIG. 17. In this case, the block is laid at a positionshifted to the right by two pixels from the position of the input data.

As described above, when motion image data is reproduced from datasubjected to the decimation process in which the sampling point positionwas shifted in the horizontal direction, the mixer 330 horizontallyshifts, frame by frame, the positions at which reproduced blocks arelaid. The reason why shifting of positions is necessary is describedbelow with reference to FIG. 50.

FIGS. 50A to 50D each show pixels constituting part of a block of framedata corresponding to FIGS. 49A to 49D. As can be seen from FIGS. 50A to50D, in the production of conversion data, the sampling point positionwas shifted by one pixel to the right, frame by frame.

When decimation is performed by factor m=4, four pixel values in ahorizontal line are decimated into one pixel value equal to one of thefour pixel values. However, the true position of such the pixel valuechanges by one pixel, frame by frame, as shown in FIGS. 50A to 50Drather than at a fixed position. If the position of the pixel value isset to be nearly at the center of the display area, then the pixelposition changes in the display area (denoted by a dotted line) of theoriginal data starting from a position shifted to the left by one pixelshown in FIG. 50A to a position shifted to the right by two pixels shownin FIG. 50D, and thus the resultant displayed image becomes closer tothe original image data.

FIGS. 51A to 51D show examples of manners of laying blocks supplied fromone of the block decompressor 321 to 323 in a case in which the blockswere subjected to the spatial decimation in which the sampling pointposition was shifted vertically.

The block shown in FIG. 51A is a block of a first frame (when motionimage data is processed in units of 4 (=N) frames) reproduced from blockdata that was subjected to the decimation in which the sampling pointposition was shifted downward with frame advance, in the manner asdescribed earlier with reference to FIG. 25, or the block shown in FIG.51A is a block of a fourth frame (when motion image data is processed inunits of 4 (=N) frames) reproduced from block data that was subjected tothe decimation in which the sampling point position was shifted upwardwith frame advance, in the manner as described earlier with reference toFIG. 26. In this case, the block is laid at a position shifted upward byone pixel from the position of the input data.

The block shown in FIG. 51B is a block of a second frame (when motionimage data is processed in units of 4 (=N) frames) reproduced from blockdata that was subjected to the decimation in which the sampling pointposition was shifted downward with frame advance, in the manner asdescribed earlier with reference to FIG. 25, or the block shown in FIG.51B is a block of a third frame (when motion image data is processed inunits of 3 (=N) frames) reproduced from block data that was subjected tothe decimation in which the sampling point position was shifted upwardwith frame advance, in the manner as described earlier with reference toFIG. 26. In this case, the block is laid at the same position as theposition of the input data.

The block shown in FIG. 51C is a block of a third frame (when motionimage data is processed in units of 4 (=N) frames) reproduced from blockdata that was subjected to the decimation in which the sampling pointposition was shifted downward with frame advance, in the manner asdescribed earlier with reference to FIG. 25, or the block shown in FIG.51C is a block of a second frame (when motion image data is processed inunits of 2 (=N) frames) reproduced from block data that was subjected tothe decimation in which the sampling point position was shifted upwardwith frame advance, in the manner as described earlier with reference toFIG. 26. In this case, the block is laid at a position shifted downwardby one pixel from the position of the input data.

The block shown in FIG. 51D is a block of a fourth frame (when motionimage data is processed in units of 4 (=N) frames) reproduced from blockdata that was subjected to the decimation in which the sampling pointposition was shifted downward with frame advance, in the manner asdescribed earlier with reference to FIG. 25, or the block shown in FIG.51D is a block of a first frame (when motion image data is processed inunits of 4 (=N) frames) reproduced from block data that was subjected tothe decimation in which the sampling point position was shifted upwardwith frame advance, in the manner as described earlier with reference toFIG. 26. In this case, the block is laid at a position shifted downwardby two pixels from the position of the input data.

As described above, when motion image data is reproduced from datasubjected to the decimation process in which the sampling point positionwas shifted vertically, the mixer 330 vertically shifts, frame by frame,the positions at which reproduced blocks are laid for a reason similarto the reason for the horizontal shifting described above with referenceto FIG. 50, that is, the vertical shifting is performed to achieve areproduced image more similar to the original image.

When a block is laid at a position shifted from an original position asdescribed above, there is a possibility that two adjacent blocks overlapeach other as shown in FIG. 52A or a gap is created between two adjacentblocks as shown in FIG. 52B, depending on the manner in which adjacenttwo blocks was processed. When there is an overlap, pixel values in theoverlap area may be replaced with the average values of pixel values ofrespective blocks in the overlap area. When a gap is created, the gapmay be filled with pixel values calculated, for example, by means oflinear interpolation, from pixel values located at edges facing the gap.After obtaining a complete frame data by performing such a correction,the combiner 330 outputs the resultant frame data.

The present invention has been described above with reference tospecific embodiments by way of example and not limitation. It should beapparent to those skilled in the art that various modifications andsubstitutions are possible without departing from the spirit and thescope of the invention. That is, the embodiments have been describedabove by way of example and not limitation. The scope of the inventionis to be determined solely by the claims.

Any of the processes disclosed in the present description may beperformed by means of hardware, software, or a combination of hardwareand software. In the case in which a process is performed by means ofsoftware, a program of the process may be installed into a memorydisposed in a dedicated computer embedded in hardware and the programmay be executed by the computer, or the program may be installed on ageneral-purpose computer capable of executing various processes and maybe executed on the general-purpose computer.

The program may be stored in advance in a storage medium such as a harddisk or a ROM (Read Only Memory). The program may also be temporarily orpermanently stored in a removable storage medium such as a flexibledisk, a CD-ROM (Compact Disc Read Only Memory), an MO (Magneto-optical)disk, a DVD (Digital Versatile Disc), a magnetic disk, or asemiconductor memory. The program stored on such a removable storagemedium may be supplied in the form of so-called packaged software.

Instead of installing the program from the removable storage medium ontothe computer, the program may also be transferred to the computer from adownload site via radio transmission or via a network such as an LAN(Local Area Network) or the Internet by means of wire communication. Inthis case, the computer receives the program transmitted in theabove-described manner and installs the program on a storage medium suchas a hard disk disposed in the computer.

The processes disclosed in the present description may be performedtime-sequentially in the same order as that described in the program, ormay be performed in parallel or individually depending on the processingpower of the computer. In the present description, the term “system” isused to describe a logical collection of a plurality of devices, and itis not necessarily required that the plurality of devices be disposed ina single case.

It should be understood by those skilled in the art that variousmodifications, combinations, sub-combinations and alterations may occurdepending on design requirements and other factors insofar as they arewithin the scope of the appended claims or the equivalents thereof.

1. A motion image reproduction apparatus that reproduces motion imagedata from converted motion image data, comprising: a block expander thatreceives converted block data included in the converted motion imagedata and data indicating a conversion mode in which the converted blockdata was converted and that expands the converted block data based onthe data indicating the conversion mode; and a combiner that producesframe data by combining blocks reproduced via the block expansionperformed by the block expander, wherein, the block expander reproducesthe block by performing the block expansion process in a modecorresponding to at least one of a spatial decimation process in a fixedsampling point position mode, a spatial decimation process in a samplingpoint position shifting mode, and a temporal decimation process,performed in the production of the converted motion image data, and thecombiner produces the frame data by combining blocks reproduced via theblock expansion process.
 2. A motion image reproduction apparatusaccording to claim 1, wherein when the blocks reproduced by the blockexpander are blocks reproduced from blocks subjected to the spatialdecimation process in the sampling point position shifting mode, and thecombiner shifts the positions at which the blocks are laid with frameadvance.
 3. A motion image reproduction apparatus according to claim 1,wherein when the blocks reproduced by the block expander are blocksreproduced from blocks subjected to the spatial decimation process inthe sampling point position shifting mode, the combiner shifts thepositions at which the blocks are laid with frame advance and performs apixel value correction process to determine a pixel value in a pixel gapcreated when blocks were laid or determine a pixel value for anoverlapping pixel.
 4. A method of reproducing motion image data fromconverted motion image data, comprising the steps of: expanding a blockby receiving converted block data included in the converted motion imagedata and data indicating a conversion mode in which the converted blockdata was converted, and expanding the converted block data based on thedata indicating the conversion mode; and combining blocks reproduced inthe block expansion step to produce frame data, wherein, the blockexpansion step reproduces the block by performing the block expansionprocess in a mode corresponding to at least one of a spatial decimationprocess in a fixed sampling point position mode, a spatial decimationprocess in a sampling point position shifting mode, and a temporaldecimation process, performed in the production of the converted motionimage data, and the block combining step produces the frame data bycombining blocks reproduced via the block expansion process.
 5. A methodof reproducing motion image data, according to claim 4, wherein in theblock combining step, when the blocks reproduced by the block expanderare blocks reproduced from blocks subjected to the spatial decimationprocess in the sampling point position shifting mode, the positions atwhich to lay the blocks are shifted with frame advance.
 6. A method ofreproducing motion image data, according to claim 4, wherein the blockcombining step includes the step of, when the blocks reproduced by theblock expander are blocks reproduced from blocks subjected to thespatial decimation process in the sampling point position shifting mode,shifting the positions at which the blocks are laid with frame advanceand performing a pixel value correction process to determine a pixelvalue in a pixel gap created when blocks were laid or determine a pixelvalue for an overlapping pixel.
 7. A computer program for executing aprocess of reproducing motion image data from converted motion imagedata, the process comprising the steps of: expanding a block byreceiving converted block data included in the converted motion imagedata and data indicating a conversion mode in which the converted blockdata was converted, and expanding the converted block data based on thedata indicating the conversion mode; and combining blocks reproduced inthe block expansion step to produce frame data, wherein, the blockexpansion step reproduces the block by performing the block expansionprocess in a mode corresponding to at least one of a spatial decimationprocess in a fixed sampling point position mode, a spatial decimationprocess in a sampling point position shifting mode, and a temporaldecimation process, performed in the production of the converted motionimage data, and the block combining step produces the frame data bycombining blocks reproduced via the block expansion process.